[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235103#comment-17235103
]
angerszhu commented on YARN-9561:
-
[~ebadger] we build hadoop with
{code:java}
-Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
{code}
Under jenkins machine /usr/lib/x86_64-linux-gun
{code:java}
-rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
-rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
-rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so -> libcrypto.so.1.0.0
-rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
/lib/x86_64-linux-gnu/libcrypt.so.1
-rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
{code}
Under nodemanager /usr/lib/x86_64-linux-gun
{code:java}
-rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
-rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so -> libcrypto.so.1.1
-rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
-rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
/lib/x86_64-linux-gnu/libcrypt.so.1
-rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
{code}
The libcrypto.so 's version is not same.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch,
> YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch,
> YARN-9561.015.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235102#comment-17235102
]
angerszhu commented on YARN-9561:
-
[~ebadger] we build hadoop with
{code:java}
-Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
{code}
Under jenkins machine /usr/lib/x86_64-linux-gun
!image-2020-11-19-09-43-54-073.png!
Under nodemanager /usr/lib/x86_64-linux-gun
!image-2020-11-19-09-44-11-200.png!
The libcrypto.so 's version is not same.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch,
> YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch,
> YARN-9561.015.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234867#comment-17234867
]
Eric Badger commented on YARN-9561:
---
I think the short answer is that you need to have openssl-devel installed on
the machine where you are compiling Hadoop.
>From BUILDING.txt
{noformat}
160 * Use -Drequire.openssl to fail the build if libcrypto.so is not found.
161 If this option is not specified and the openssl library is missing,
162 we silently build a version of libhadoop.so that cannot make use of
163 openssl. This option is recommended if you plan on making use of openssl
164 and want to get more repeatable builds.
165 * Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto
166 header files and library files. You do not need this option if you have
167 installed openssl using a package manager.
168 * Use -Dopenssl.lib to specify a nonstandard location for the libcrypto
library
169 files. Similarly to openssl.prefix, you do not need this option if you
have
170 installed openssl using a package manager.
171 * Use -Dbundle.openssl to copy the contents of the openssl.lib directory
into
172 the final tar file. This option requires that -Dopenssl.lib is also
given,
173 and it ignores the -Dopenssl.prefix option. If -Dopenssl.lib isn't
given, the
174 bundling and building will fail.
{noformat}
The crypto library is statically linked to the container-executor during
compilation. I guess it just quietly moves on if it doesn't find it instead of
failing. That's sort of troubling and something I'll look into fixing. But the
answer is to make sure openssl-devel is installed before you compile Hadoop
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch,
> YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch,
> YARN-9561.015.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234460#comment-17234460
]
angerszhu commented on YARN-9561:
-
When we upgrade from hadoop-3.2.1 to hadoop-3.3.0, we can't start nodemanager
{code:java}
... 3 more
Caused by:
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
object file: No such file or directory
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
... 4 more
Caused by: ExitCodeException exitCode=127:
/home/hadoop/hadoop/bin/container-executor: error while loading shared
libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or
directory
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
... 6 more
{code}
Seems related to this patch and we know that it problem that it can't find path
to libcrypto.so.1.0.0, seems $HADOOP_HOME/lib/native won't under
container-executor's RPATH.
Can anyone give some advise about how to fix this problem elegantly
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch,
> YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch,
> YARN-9561.015.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991054#comment-16991054 ] Hudson commented on YARN-9561: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17740 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17740/]) Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. (ebadger: rev ab2cc5ac92df21fa391d9f480ae4d8dca7f6cefd) * (edit) LICENSE.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_main.cc YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed (ebadger: rev 8dffd8dc8938a59f42427ab787b38693719b07f7) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_base_ctx.h * (edit) LICENSE.txt * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_base_ctx.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_config.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/file-utils.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_main.cc * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/cJSON/cJSON.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_launch_cmd.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/file-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_write_config.h * (add) hadoop-yarn-project/hadoop-yarn/ha
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990089#comment-16990089 ] Hudson commented on YARN-9561: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17734 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17734/]) YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed (ebadger: rev 76bb297db9543eb66eee9d9306b7172378efdbb5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_main.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c * (edit) LICENSE.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch, > YARN-9561.015.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989383#comment-16989383 ] Eric Yang commented on YARN-9561: - +1 for patch 15. Thank you [~ebadger]. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch, > YARN-9561.015.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986353#comment-16986353 ] Eric Badger commented on YARN-9561: --- [~shaneku...@gmail.com], [~eyang], hope the holidays have treated you well. When you get a chance, could you check out the latest patch that fixes the Hadoop QA maven issue? > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch, > YARN-9561.015.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982778#comment-16982778
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
61m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 32s{color} |
{color:red} root generated 4 new + 22 unchanged - 4 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 42s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.security.token.delegation.TestZKDelegationTokenSecretManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12986713/YARN-9561.015.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux cca9e906ad59 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 52e9ee3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25224/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25224/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25224/testReport/ |
| Max. process+thread count | 1485 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25224/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
>
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982720#comment-16982720 ] Jim Brennan commented on YARN-9561: --- Thanks for the update [~ebadger]! I downloaded patch 015 and then verified I could build nodemanager after doing a clean at the top level. I then ran cetest and test-container-executor. I then verified I could build from the top level as well. +1 (non-binding) on patch 015. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch, > YARN-9561.015.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982003#comment-16982003
]
Eric Badger commented on YARN-9561:
---
Attaching patch 015 that removes the dependency on libhadoop and adds the
source files directly into the compilation in CMakeLists.txt. I'm not sure that
this is the correct solution, but it does solve the issue. that was seen in
YARN-9988.
Thanks to [~Jim_Brennan] for finding this solution and testing it out.
{noformat}
ebadger@C02XG1MYJGH8: diff ~/patches/YARN-9561.014.patch
~/patches/YARN-9561.015.patch
33c33
< index 7e8b19f3e66..92e17976e72 100644
---
> index 7e8b19f3e66..677429bb99c 100644
36c36
< @@ -16,7 +16,8 @@
---
> @@ -16,12 +16,15 @@
46c46,53
< @@ -115,6 +116,7 @@ include_directories(
---
> set(GTEST_SRC_DIR
> ${CMAKE_SOURCE_DIR}/../../../../../hadoop-common-project/hadoop-common/src/main/native/gtest)
>
> +set(HADOOP_COMMON_SEC_PATH
> ${HADOOP_COMMON_PATH}/src/main/native/src/org/apache/hadoop/security)
> +
> # determine if container-executor.conf.dir is an absolute
> # path in case the OS we're compiling on doesn't have
> # a hook in get_executable. We'll use this define
> @@ -115,6 +118,7 @@ include_directories(
50c57
< +${HADOOP_COMMON_PATH}/src/main/native/src/org/apache/hadoop/security
---
> +${HADOOP_COMMON_SEC_PATH}
54c61
< @@ -129,6 +131,7 @@ add_library(container
---
> @@ -129,6 +133,7 @@ add_library(container
62c69
< @@ -138,6 +141,12 @@ add_library(container
---
> @@ -138,6 +143,14 @@ add_library(container
71a79,80
> +${HADOOP_COMMON_SEC_PATH}/hadoop_user_info.c
> +${HADOOP_COMMON_SEC_PATH}/hadoop_group_info.c
75c84
< @@ -146,6 +155,8 @@ add_executable(container-executor
---
> @@ -146,6 +159,7 @@ add_executable(container-executor
79d87
< +${HADOOP_COMMON_PATH}/target/native/target/usr/local/lib/libhadoop.a
84c92
< @@ -155,7 +166,10 @@ add_executable(test-container-executor
---
> @@ -155,7 +169,9 @@ add_executable(test-container-executor
91d98
< +${HADOOP_COMMON_PATH}/target/native/target/usr/local/lib/libhadoop.a
96c103
< @@ -173,8 +187,16 @@ add_executable(cetest
---
> @@ -173,8 +189,15 @@ add_executable(cetest
108d114
< +${HADOOP_COMMON_PATH}/target/native/target/usr/local/lib/libhadoop.a
{noformat}
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch,
> YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch,
> YARN-9561.015.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981945#comment-16981945
]
Eric Badger commented on YARN-9561:
---
[~eyang], [~shaneku...@gmail.com], [~ccondit], do you guys know how to add
libhadoop as a dependency for the container-executor? I'm not sure if this
needs to be done via CMakeLists.txt or a pom.xml. Right now, the compilation
code assumes that libhadoop.a exists. So obviously the problem is that the
hadoop-yarn-server-nodemanager subproject doesn't have the dependency correctly
formed to compile libhadoop if it doesn't already exist.
{noformat:title=hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml}
38
39 org.apache.hadoop
40 hadoop-common
41 provided
42
{noformat}
There is a dependency for hadoop-common, but it is {{provided}}. I tried
changing it to {{compile}}, but that doesn't work.
{noformat:title=hadoop-common-project/hadoop-common/src/CMakeLists.txt}
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
hadoop_add_dual_library(hadoop
main/native/src/exception.c
${SRC}/io/compress/lz4/Lz4Compressor.c
${SRC}/io/compress/lz4/Lz4Decompressor.c
${SRC}/io/compress/lz4/lz4.c
${SRC}/io/compress/lz4/lz4hc.c
${ISAL_SOURCE_FILES}
${SNAPPY_SOURCE_FILES}
${ZSTD_SOURCE_FILES}
${OPENSSL_SOURCE_FILES}
${SRC}/io/compress/zlib/ZlibCompressor.c
${SRC}/io/compress/zlib/ZlibDecompressor.c
${BZIP2_SOURCE_FILES}
${SRC}/io/nativeio/NativeIO.c
${PMDK_SOURCE_FILES}
${SRC}/io/nativeio/errno_enum.c
${SRC}/io/nativeio/file_descriptor.c
${SRC}/io/nativeio/SharedFileDescriptorFactory.c
${SRC}/net/unix/DomainSocket.c
${SRC}/net/unix/DomainSocketWatcher.c
${SRC}/security/JniBasedUnixGroupsMapping.c
${SRC}/security/JniBasedUnixGroupsNetgroupMapping.c
${SRC}/security/hadoop_group_info.c
${SRC}/security/hadoop_user_info.c
${SRC}/util/NativeCodeLoader.c
${SRC}/util/NativeCrc32.c
${SRC}/util/bulk_crc32.c
${BULK_CRC_ARCH_SOURCE_FIlE}
)
if(NEED_LINK_DL)
set(LIB_DL dl)
endif()
hadoop_target_link_dual_libraries(hadoop ${LIB_DL} ${JAVA_JVM_LIBRARY})
set(LIBHADOOP_VERSION "1.0.0")
set_target_properties(hadoop PROPERTIES SOVERSION ${LIBHADOOP_VERSION})
hadoop_dual_output_directory(hadoop target/usr/local/lib)
# By embedding '$ORIGIN' into the RPATH of libhadoop.so, dlopen will look in
# the directory containing libhadoop.so. However, $ORIGIN is not supported by
# all operating systems.
if(${CMAKE_SYSTEM_NAME} MATCHES "Linux|SunOS")
set(RPATH "\$ORIGIN/")
if(EXTRA_LIBHADOOP_RPATH)
set(RPATH "${RPATH}:${EXTRA_LIBHADOOP_RPATH}/")
endif()
set_target_properties(hadoop PROPERTIES INSTALL_RPATH "${RPATH}")
endif()
{noformat}
libhadoop seems to be getting built by these lines in the hadoop-common
CMakeLists.txt. I tried simply adding {{hadoop}} instead of the full
{{libhadoop.a}} path, but that also (unsurprisingly) didn't work.
I imagine there's some special incantation that will allow me to add libhadoop
as a dependency, but I'm not sure how to do it or at what level (maven or
CMake) to do it. Any help would be appreciated.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch,
> YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981770#comment-16981770 ] Hudson commented on YARN-9561: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17692 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17692/]) Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. (ebadger: rev 6c20512b6897c444591d2a90fdded087d1aa16f7) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_reap.h * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_launch_cmd.c * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/cJSON/cJSON.h * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_base_ctx.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_write_config.h * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/cJSON/cJSON.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * (edit) LICENSE.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_main.cc * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_config.h * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_write_config.c * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.h * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/file-utils.h * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_reap.c * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/file-utils.c * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_base_ctx.c * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_launch_cmd.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.h > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YA
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976937#comment-16976937 ] Hudson commented on YARN-9561: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17658 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17658/]) YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed (ebadger: rev 289bbca8709e17f962f608154d0bfbd617a69297) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_write_config.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/file-utils.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/cJSON/cJSON.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_base_ctx.c * (edit) LICENSE.txt * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/cJSON/cJSON.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/file-utils.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_base_ctx.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_launch_cmd.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_launch_cmd.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_config.h * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_reap.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_reap.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_main.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/runc/runc_write_config.c * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, Y
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975247#comment-16975247 ] Jim Brennan commented on YARN-9561: --- I'm +1 (non-binding) on patch 014. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975236#comment-16975236 ] Craig Condit commented on YARN-9561: I'm +1 on this one (non-binding). LGTM. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975230#comment-16975230 ] Eric Badger commented on YARN-9561: --- [~shaneku...@gmail.com], could you also +1 YARN-9562 if you think it's good to go? [~eyang], [~ccondit], I'll commit this later today unless there is objection. If you would like more time to review, let me know. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16975061#comment-16975061 ] Shane Kumpf commented on YARN-9561: --- I retested without the private tmp mount and moving my YARN local dirs out of tmp, both will resolve the issue I previously saw around clean up. +1 from me. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974472#comment-16974472 ] Eric Badger commented on YARN-9561: --- Patch 014 addresses the const issue brought up by cc > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch, YARN-9561.014.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973946#comment-16973946
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
64m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 36s{color} |
{color:red} root generated 4 new + 23 unchanged - 3 fixed = 27 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}166m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}296m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985803/YARN-9561.013.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 3a6635f772a4 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 73a386a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25165/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25165/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25165/testReport/ |
| Max. process+thread count | 2895 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25165/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the ne
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973850#comment-16973850 ] Eric Badger commented on YARN-9561: --- Patch 013 updates the error logic around creating private tmp. [~shaneku...@gmail.com] also discovered a bug around mounting anything onto /tmp or /var/tmp along with this private tmp feature. We will deal with that in YARN-9959 > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch, YARN-9561.013.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973694#comment-16973694
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
61m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 14m 45s{color} |
{color:red} root generated 6 new + 21 unchanged - 5 fixed = 27 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 49s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}262m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
| | hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985549/YARN-9561.012.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 868d67dbe031 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / df6b316 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25160/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25160/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25160/testReport/ |
| Max. process+thread count | 4329 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25160/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C chan
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971972#comment-16971972
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
63m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 35s{color} |
{color:red} root generated 4 new + 23 unchanged - 3 fixed = 27 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}157m 54s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
3s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}287m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985549/YARN-9561.012.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux bc21c94d0655 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 30b93f9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25140/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25140/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25140/testReport/ |
| Max. process+thread count | 3171 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25140/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> --
>
>
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971846#comment-16971846 ] Eric Badger commented on YARN-9561: --- [~eyang], looks like there was a bug there beacuse I wasn't explicitly setting up the container-executor.cfg file in the cetest framework before setting the nm user. That should be fixed in patch 012, which I have attached. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch, > YARN-9561.012.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970666#comment-16970666
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 20m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
76m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 19m 16s{color} |
{color:red} root generated 4 new + 22 unchanged - 4 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 19m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}171m 56s{color}
| {color:red} root in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} asflicense {color} | {color:blue} 0m
48s{color} | {color:blue} ASF License check generated no output? {color} |
| {color:black}{color} | {color:black} {color} | {color:black}321m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestDisableConnCache |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.TestErasureCodingExerciseAPIs |
| | hadoop.hdfs.TestByteBufferPread |
| | hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy |
| | hadoop.hdfs.TestParallelShortCircuitReadNoChecksum |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.TestHAAuxiliaryPort |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.TestFileCreationEmpty |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.TestErasureCodingAddConfig |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.TestSetrepDecreasing |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.TestDFSShellGenericOptions |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.TestStateAlignmentContextWithHA |
| | hadoop.hdfs.TestDistributedFileSystemWithECFile |
| | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.client.impl.TestClientBlockVerification |
| |
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985387/YARN-9561.011
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970646#comment-16970646
]
Eric Yang commented on YARN-9561:
-
[~ebadger] What is the right way to run test_runc_util with patch 11? Cetest
is crashing on my machine:
{code}
mvn clean test -Dtest=cetest -Pnative
{code}
Maven output looks like this:
{code}
[INFO] ---
[INFO] C M A K E B U I L D E RT E S T
[INFO] ---
[INFO] cetest: running
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest
--gtest_filter=-Perf.
--gtest_output=xml:/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports/TEST-cetest.xml
[INFO] with extra environment variables {}
[INFO] STATUS: ERROR CODE 139 after 5 millisecond(s).
[INFO] ---
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 01:05 min
[INFO] Finished at: 2019-11-08T18:29:30-05:00
[INFO] Final Memory: 56M/575M
[INFO]
[ERROR] Failed to execute goal
org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:cmake-test (cetest) on
project hadoop-yarn-server-nodemanager: Test
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest
returned ERROR CODE 139 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
{code}
Run the test manually reveals:
{code}
$ ./cetest
Determining user details
Requested user eyang is not whitelisted and has id 501,which is below the
minimum allowed 1000
Setting NM UID
Segmentation fault
{code}
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970535#comment-16970535 ] Eric Badger commented on YARN-9561: --- Thanks for the review, [~Jim_Brennan]! bq. stat_file_as_nm should ensure that it restores the calling user/group before returning. As we discussed offline, the check does work as designed. bq. (nit) might want to change the name of stat_file_as_nm - it is not at all clear from the name that it will fail if the file exists and succeed if it doesn't Good call. I updated the name in patch 011. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch, YARN-9561.011.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970380#comment-16970380 ] Jim Brennan commented on YARN-9561: --- Thanks for the updates [~ebadger]! A couple comments on the new patch: * stat_file_as_nm should ensure that it restores the calling user/group before returning. * (nit) might want to change the name of stat_file_as_nm - it is not at all clear from the name that it will fail if the file exists and succeed if it doesn't > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch, YARN-9561.010.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969732#comment-16969732
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
66m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 16m 26s{color} |
{color:red} root generated 3 new + 23 unchanged - 3 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}166m 10s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}300m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985269/YARN-9561.010.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 085fee6cfba8 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 247584e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25116/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25116/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25116/testReport/ |
| Max. process+thread count | 2752 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25116/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> -
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969642#comment-16969642
]
Eric Badger commented on YARN-9561:
---
Patch 010 adds functionality related to the {{nobody}} user in local user mode.
Details are in YARN-9562.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch, YARN-9561.010.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967018#comment-16967018
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
66m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 16m 37s{color} |
{color:red} root generated 2 new + 24 unchanged - 2 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 36s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984827/YARN-9561.009.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux cdd7862ee066 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 51e7d1b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25090/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25090/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25090/testReport/ |
| Max. process+thread count | 1348 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25090/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966954#comment-16966954 ] Jim Brennan commented on YARN-9561: --- Thanks for fixing that [~ebadger]! The change looks good to me, and I verified that it works for me with container-executor.conf.dir set. I am +1 (non-binding) on patch 009. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch, > YARN-9561.009.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966917#comment-16966917
]
Eric Badger commented on YARN-9561:
---
Thanks for the review, [~Jim_Brennan]! Patch 009 changes up
{{create_ce_file()}} a little bit to do stat checking beforehand and only
creating the file if it doesn't already exist.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch,
> YARN-9561.009.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965064#comment-16965064 ] Jim Brennan commented on YARN-9561: --- Actually, it might be better in this case to just do the stat and fail if it doesn't exist. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965061#comment-16965061
]
Jim Brennan commented on YARN-9561:
---
Thanks for updating the patch [~ebadger]! I tested this along with the patches
for YARN-9562 and YARN-9564. Everything seems to be working well. I did run
into one issue with the container executor unit tests (cetest). I normally
compile with this option: -Dcontainer-executor.conf.dir=${HADOOP_CONF_DIR}
This causes some failures in cetest:
{noformat}
[--] 7 tests from TestRunc
[ RUN ] TestRunc.test_parse_runc_launch_cmd_valid
Could not create /home/gs/hadoop/conf/container-executor.cfg
/home/jbrennan02/git/y-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc:47:
Failure
Expected: ret
Which is: 1
To be equal to: 0
Container executor cfg setup failed
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_valid (1 ms)
[ RUN ] TestRunc.test_parse_runc_launch_cmd_bad_container_id
Could not create /home/gs/hadoop/conf/container-executor.cfg
/home/jbrennan02/git/y-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc:47:
Failure
Expected: ret
Which is: 1
To be equal to: 0
Container executor cfg setup failed
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_bad_container_id (0 ms)
[ RUN ] TestRunc.test_parse_runc_launch_cmd_existing_pidfile
Could not create /home/gs/hadoop/conf/container-executor.cfg
/home/jbrennan02/git/y-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc:47:
Failure
Expected: ret
Which is: 1
To be equal to: 0
Container executor cfg setup failed
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_existing_pidfile (0 ms)
[ RUN ] TestRunc.test_parse_runc_launch_cmd_invalid_media_type
Could not create /home/gs/hadoop/conf/container-executor.cfg
/home/jbrennan02/git/y-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc:47:
Failure
Expected: ret
Which is: 1
To be equal to: 0
Container executor cfg setup failed
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_invalid_media_type (0 ms)
[ RUN ] TestRunc.test_parse_runc_launch_cmd_invalid_num_reap_layers_keep
Could not create /home/gs/hadoop/conf/container-executor.cfg
/home/jbrennan02/git/y-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc:47:
Failure
Expected: ret
Which is: 1
To be equal to: 0
Container executor cfg setup failed
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_invalid_num_reap_layers_keep
(0 ms)
[ RUN ] TestRunc.test_parse_runc_launch_cmd_valid_mounts
Could not create /home/gs/hadoop/conf/container-executor.cfg
/home/jbrennan02/git/y-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc:47:
Failure
Expected: ret
Which is: 1
To be equal to: 0
Container executor cfg setup failed
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_valid_mounts (0 ms)
[ RUN ] TestRunc.test_parse_runc_launch_cmd_invalid_mounts
Could not create /home/gs/hadoop/conf/container-executor.cfg
/home/jbrennan02/git/y-hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_runc_util.cc:47:
Failure
Expected: ret
Which is: 1
To be equal to: 0
Container executor cfg setup failed
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_invalid_mounts (0 ms)
[--] 7 tests from TestRunc (2 ms total)
[--] Global test environment tear-down
[==] 89 tests from 10 test cases ran. (82 ms total)
[ PASSED ] 82 tests.
[ FAILED ] 7 tests, listed below:
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_valid
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_bad_container_id
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_existing_pidfile
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_invalid_media_type
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_invalid_num_reap_layers_keep
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_valid_mounts
[ FAILED ] TestRunc.test_parse_runc_launch_cmd_invalid_mounts
{noformat}
It's failing because I already have a container-executor.cfg there and it is
owned by root.
If I run without defining {{container-executor.conf.dir}}, all of the tests
pass.
I was able to get this to work by modifying
test_runc_util.cc::create_ce_file():
{noformat}
int create_ce_f
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963517#comment-16963517
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
63m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 14m 53s{color} |
{color:red} root generated 6 new + 20 unchanged - 6 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 59s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}268m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984401/YARN-9561.008.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 45084a91bbe3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9a2e43e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25068/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25068/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25068/testReport/ |
| Max. process+thread count | 4072 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25068/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> --
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963334#comment-16963334 ] Eric Badger commented on YARN-9561: --- I have attached patch 008 to address some review comments of [~Jim_Brennan]'s from YARN-9562. I also found and fixed a bug in the error handling where the container-executor wouldn't fail on bad mounts. The patches that work together. YARN-9562.011.patch YARN-9561.008.patch YARN-9564.004.patch > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch, YARN-9561.008.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961705#comment-16961705
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
68m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 41s{color} |
{color:red} root generated 4 new + 22 unchanged - 4 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}231m 20s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}365m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterFaultTolerant |
| | hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload |
| | hadoop.hdfs.server.federation.metrics.TestMetricsBase |
| | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
| |
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities |
| |
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
|
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
|
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
|
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
| | hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
|
| | hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown |
| | hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps |
| | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984048/YARN-9561.007.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 249e6fc1c02c 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | mave
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961070#comment-16961070
]
Eric Badger commented on YARN-9561:
---
{noformat}
[WARNING]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/inotify.pb.cc:467:13:
warning: 'dynamic_init_dummy_inotify_2eproto' defined but not used
[-Wunused-variable]
[WARNING]
/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/erasurecoding.pb.cc:683:13:
warning: 'dynamic_init_dummy_erasurecoding_2eproto' defined but not used
[-Wunused-variable]
{noformat}
Not sure why hadoopqa is noting new cc warnings. These are the errors listed,
but they are clearly not from code that this patch touches.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch,
> YARN-9561.006.patch, YARN-9561.007.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960229#comment-16960229
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
73m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 17m 10s{color} |
{color:red} root generated 2 new + 24 unchanged - 2 fixed = 26 total (was 26)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}156m 33s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}300m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.TestMultipleNNPortQOP |
| | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12984048/YARN-9561.007.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux fe71e223b228 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / eef34f2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25044/artifact/out/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25044/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25044/testReport/ |
| Max. process+thread count | 3058 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25044/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new Ru
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960011#comment-16960011 ] Eric Badger commented on YARN-9561: --- Attaching patch 007 to fix a lack of test failures on container executor cfg setup failure > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch, YARN-9561.007.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959344#comment-16959344
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
2s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m
54s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 31m
39s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 39s{color} |
{color:red} root generated 9 new + 17 unchanged - 4 fixed = 26 total (was 21)
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 39s{color}
| {color:red} root generated 893 new + 999 unchanged - 0 fixed = 1892 total
(was 999) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}143m 27s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
5s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}241m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| | hadoop.yarn.server.webproxy.amfilter.TestAmFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.4 Server=19.03.4 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983960/YARN-9561.006.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 2e6103369e16 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a1b4eeb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/25041/artifact/out/branch-compile-root.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-YARN-Build/25041/artifact/out/branch-mvnsite-root.txt
|
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/25041/artifact/out/diff-compile-cc-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/25041/artifact/out/diff-compile-javac-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25041/artifact/out/patch-unit-root.txt
|
| Test Results |
http
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959223#comment-16959223 ] Eric Badger commented on YARN-9561: --- Thanks for the review, [~Jim_Brennan]! Attached patch 006 to address your comments > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch, > YARN-9561.006.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959191#comment-16959191
]
Jim Brennan commented on YARN-9561:
---
Thanks for the updated patch [~ebadger]!
I had a few comments on patch 005:
runc_write_config()
* add_std_mounts_json() - should we increase shm to 8GB like we did
internally?
main.c
- [BUG] main() - missing {{break}} statement after
{{RUN_AS_USER_SYNC_YARN_SYSFS}} block.
test_string_utils.cc
- TEST_F(TestStringUtils, test_strbuf_detach)
this test would be a little better if it moved the buf contents assert to after
the third append format and/or included a check that sb.buffer != buf (after
that last append).
- TEST_F(TestStringUtils, test_strbuf_realloc)
looks like this has some left over debug std::cout lines?
test_runc_util.cc
- build_process_struct()
returns true when is fails and false when it succeeds?
- build_mounts_json()
is there a reason for three options string arrays? The first two are
identical. Can’t you just have options_ro and options_rw?
- (nit) unindented line: {{remove(pid_file);}}
- Would be nice to move these lines into a function - they are repeated a lot:
{noformat}
std::string container_executor_cfg_contents = "[runc]\n "
"runc.allowed.rw-mounts=/opt,/var,/usr/bin/cut,/usr/bin/awk\n "
"runc.allowed.ro-mounts=/etc/passwd";
ret = setup_container_executor_cfg(container_executor_cfg_contents);
ASSERT_EQ(ret, 0) << "Container executor cfg setup failed\n"; {noformat}
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959002#comment-16959002 ] Jim Brennan commented on YARN-9561: --- [~ebadger] it does not look like Hadoop QA tests have run since you updated patch 005? > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954989#comment-16954989 ] Eric Badger commented on YARN-9561: --- Patch 005 is based on YARN-9884 patch 004. Together with YARN-9562 patch 009 and YARN-9564 patch 004 this creates a fully working RuncContainerRuntime YARN-9884.004.patch YARN-9561.005.patch YARN-9562.009.patch YARN-9564.004.patch > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954988#comment-16954988
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} YARN-9561 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12983476/YARN-9561.005.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25019/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch, YARN-9561.005.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924470#comment-16924470
]
Eric Yang commented on YARN-9561:
-
{quote}There's currently a debug line at the end of
RuncContainerRuntime.launchContainer() that will get printed to the nodemanager
that has the entire JSON that was used to launch the container-executor. I
didn't want to print this to info since the JSON is very large and scary
looking. What do you think?{quote}
I was thinking to print the config.json instead of runc-config.json. This
provide ability to verify that no incorrect syntax was passed onto runc. If
this is too wordy, a line of the current stage of the execution would help to
know that runc command is called.
{quote}I'm confused as to what you're asking here. In this comment on YARN-9560
you didn't want the configs to be mapped together. Now it sounds like you want
there to be integration. Could you clarify this for me?{quote}
Direct mapping of the config key is not a good idea because the parameter value
might look different even though their purpose might be the same. For example,
YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE is to support Docker entry
point, but that concept does not exist with runc. If user does not want the
wrapper launch_container.sh and like to specify the launch command directly,
configuration mechanism defines how to control the container via either
properties or environment variable to control those behaviors. The
documentation is a list of available customizable controls that is available to
application developers. Similar set of controls needs to be developed for
application developer to configure runc containers.
{quote}
If you have specific suggestions on things that you would like to be added,
feel free to create JIRAs. Otherwise I'll create a list at some point.
{quote}
The configurable properties is the suggested list to start. Mount point is the
most important one. The current patch offers its own implementation of default
mount point. The app submission REST API offers a different way to mount
devices with acl check. I think the second approach provides better security.
What is the right way to ensure the two techniques are complementary of each
other and reduce divergence? Can someone submit a job.xml that customize
yarn.nodemanager.runtime.linux.runc.default-rw-mounts? It looks like possible
today, but should it be allowed?
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923766#comment-16923766
]
Eric Badger commented on YARN-9561:
---
bq. Eric Badger thank you for the debugging session today. I got mapreduce pi
to run correctly after adding /etc/krb5.conf to default mount location.
Fantastic news!! Really great that you were able to get it working end-to-end.
Now we just need to clean the patches up and we should be good to go.
bq. Print a line of output before calling runc. Prelaunch initialization is
completed or print the formatted json in prelaunch.out. This helps to narrow
down the root of the problem is caused by user job configuration or bugs in
container-executor code. Docker runtime shows the command line for calling
docker. This helps to troubleshoot the actual problem sooner.
There's currently a debug line at the end of
{{RuncContainerRuntime.launchContainer()}} that will get printed to the
nodemanager that has the entire JSON that was used to launch the
container-executor. I didn't want to print this to info since the JSON is very
large and scary looking. What do you think?
bq. YARN Docker integration have a list of configurable properties. These
settings do not work with runc container today. Without hinder progress, I
suggest to open new issues to improve integration.
I'm confused as to what you're asking here. In [this
comment|https://issues.apache.org/jira/browse/YARN-9560?focusedCommentId=16851429&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16851429]
on YARN-9560 you didn't want the configs to be mapped together. Now it sounds
like you want there to be integration. Could you clarify this for me?
bq. YARN service uses the properties defined in #3 for customize YARN services
mount points, network to use, and privilege container flag. Similar feature
sets need new tickets to ensure the new runtime can integrate well with YARN
service programming interfaces.
Yes, absolutely. As we talked on the docker meeting call, there are a lot of
things that will need to be added onto the runC runtime so that it can do the
same things that Docker does, and followup JIRAs are a good place for those. If
you have specific suggestions on things that you would like to be added, feel
free to create JIRAs. Otherwise I'll create a list at some point.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922888#comment-16922888
]
Eric Yang commented on YARN-9561:
-
[~ebadger] thank you for the debugging session today. I got mapreduce pi to
run correctly after adding /etc/krb5.conf to default mount location. Some
improvements to make this better:
1. The current output looks like this when container run fails:
{code}
[2019-09-04 13:26:30.726]Exception from container-launch.
Container id: container_1567624987243_0004_01_06
Exit code: 1
Exception message: Launch container failed
[2019-09-04 13:26:30.731]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
[2019-09-04 13:26:30.734]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
2019-09-04 13:26:31,455 INFO mapreduce.Job: Task Id :
attempt_1567624987243_0004_m_03_0, Status : FAILED
{code}
Print a line of output before calling runc. Prelaunch initialization is
completed or print the formatted json in prelaunch.out. This helps to narrow
down the root of the problem is caused by user job configuration or bugs in
container-executor code. Docker runtime shows the command line for calling
docker. This helps to troubleshoot the actual problem sooner.
2. ENTRY_POINT support. Instead of calling out to launch_container.sh, it
would be nice to dup the stdout, stderr without launch_container.sh wrapper.
This helps to remove the requirement of bind mounting log or workdir
directories into the container for some use cases.
3. User defined properties integration:
YARN Docker integration have a list of [configurable
properties|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/DockerContainers.html#Application_Submission].
These settings do not work with runc container today. Without hinder
progress, I suggest to open new issues to improve integration.
4. YARN service uses the properties defined in #3 for customize YARN services
mount points, network to use, and privilege container flag. Similar feature
sets need new tickets to ensure the new runtime can integrate well with YARN
service programming interfaces.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921670#comment-16921670
]
Eric Yang commented on YARN-9561:
-
[~ebadger] {quote}
I don't think this should be added in the code that was created by this patch.
The functions that are called from within setup_container_paths() are the ones
that should give more fine-grained error logging. Because at the
setup_container_paths() level I have no idea which directory failed to be
created. I would only be able to give a large list. So I think this is an
improvement for the underlying functions, but isn't relevant to this specific
patch.
{quote}
You are right that create_script_paths already provide verbose message of what
failed. I was thinking that we could print the returned parameters, but that
would be redundant to create_script_paths function.
{quote}The 16k size is just the initial size to create
upperdir=%s,workdir=%s,lowerdir=. Upperdir and workdir can only be single
directories, so the only way the 16k limit would be exceeded is if these
directory paths were absolutely gigantic. Inside of strbuf_append_fmt(){quote}
Strbuf_append_fmt is also used for recording layer information into sb
variable. Layers information may exceed 16k boundary. I am not as confident
with the buffer resize in current implementation of strbuf_append_fmt and
pointer tracking is bug free. I think there is a bug in reprint the remaining
buffer. Please provide test case for this function. Thanks
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917140#comment-16917140
]
Eric Badger commented on YARN-9561:
---
Hey [~eyang], in the meantime I got a chance to go back through your patch
comments from a little while ago. Let me know if you have any questions.
bq. In setup_container_paths, it would be good to use fprintf instead of fputs
and include the actual path location. This helps system admin to debug the
error more precisely.
I don't think this should be added in the code that was created by this patch.
The functions that are called from within {{setup_container_paths()}} are the
ones that should give more fine-grained error logging. Because at the
{{setup_container_paths()}} level I have no idea which directory failed to be
created. I would only be able to give a large list. So I think this is an
improvement for the underlying functions, but isn't relevant to this specific
patch.
bq. Can make_string be used instead of strbuf_append_fmt for readability reason
and reduce the need of string format functions? The 16k size seem like a limit
that is easy to reach. make_string may use more memory during string
construction, but maybe it is safer?
The 16k size is just the initial size to create
{{upperdir=%s,workdir=%s,lowerdir=}}. Upperdir and workdir can only be single
directories, so the only way the 16k limit would be exceeded is if these
directory paths were absolutely gigantic. Inside of {{strbuf_append_fmt()}} we
take into account how much memory needs to be allocated and reallocate that
amount. So unless the system itself is out of memory, we will be able to
allocate the full buffer regardless of the number of layers. This would be much
harder to do if we used {{make_string()}} because we would have to concatenate
the strings together or we would end up doing something like what
{{strbuf_append_fmt()}} does anyway.
bq. This part of code doesn't seem to have any effect:
{noformat}
de = is_docker_support_enabled() ? enabled : disabled;
fprintf(stream,
"%11s launch docker container: %2d appid containerid workdir "
"container-script tokens pidfile nm-local-dirs nm-log-dirs "
"docker-command-file resources ", de, LAUNCH_DOCKER_CONTAINER);
{noformat}
It has no effect on the patch related to the OCI code, yes. We fixed up the
display code to be a little bit cleaner and so we modified this to be in line
with the cleaner style.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917076#comment-16917076
]
Eric Badger commented on YARN-9561:
---
bq. Do you mean C side? Java side does not have privileges to run modprobe or
lsmod due to lack of root privileges.
I don't believe we need root privileges to run lsmod. It simply parses
/proc/modules. On RHEL 7 this file is world-readable. So I think an {{lsmod |
grep overlay}} would be sufficient.
bq. It took me several days to restore my cluster to a working state with
overlay kernel module installed. In the latest patch 004, mapreduce pi job
fails when trying to run mapreduce pi:
If you're failing in java then that means that the overlay mounts all worked
and that runC has been correctly invoked. That's fantastic news! We're very
close.
bq. Do we need implicit mounting of Hadoop binaries to enable existing workload
to run with runc? If not, what step can be used to run an example app?
I don't have any Hadoop jars in the image or bind-mounted into the image.
Instead I'm running using a Hadoop tarball in HDFS.
{noformat:title=mapred-site.xml}
mapreduce.application.framework.path
${fs.defaultFS}/user/ebadger/hadoop-3.3.0-SNAPSHOT.tar.gz#hadoop-mapreduce
mapreduce.application.classpath
./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/*,./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/*,./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/hdfs/*,./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/hdfs/lib/*,./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/*,./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/yarn/lib/*,./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/mapreduce/*,./hadoop-mapreduce/hadoop-3.3.0-SNAPSHOT/share/hadoop/mapreduce/lib/*
{noformat}
If you would like to bind-mount the hadoop jars instead, you can add them to
the default mount list
{{yarn.nodemanager.runtime.linux.docker.default-rw-mounts}} or
{{yarn.nodemanager.runtime.linux.docker.default-ro-mounts}} (don't think you
should need them to be writable). You can choose where in the image that you'd
like them to be mounted and then set your classpath up to reflect where the
jars are located.
{noformat:title=Default Mount List Example}
yarn.nodemanager.runtime.linux.docker.default-rw-mounts
/var/run/nscd:/var/run/nscd
{noformat}
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917003#comment-16917003
]
Eric Yang commented on YARN-9561:
-
[~ebadger] {quote}For your 3rd point, I think it would be better to do this
check in Java. That way we can catch the failure earlier since all containers
trying to run runC will fail if overlay is not installed. For the check, I was
thinking of doing an lsmod on "overlay"{quote}
Do you mean C side? Java side does not have privileges to run modprobe or
lsmod due to lack of root privileges.
It took me several days to restore my cluster to a working state with overlay
kernel module installed. In the latest patch 004, mapreduce pi job fails when
trying to run mapreduce pi:
{code}
vars="YARN_CONTAINER_RUNTIME_TYPE=runc,YARN_CONTAINER_RUNTIME_RUNC_IMAGE=local/java-centos:latest"
./bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0-SNAPSHOT.jar pi
-Dmapreduce.map.env=$vars -Dmapreduce.reduce.env=$vars 10 100
{code}
Observed log output:
{code}
2019-08-27 11:46:08,377 INFO mapreduce.Job: Task Id :
attempt_1566930487263_0002_m_02_2, Status : FAILED
[2019-08-27 11:46:07.131]Exception from container-launch.
Container id: container_1566930487263_0002_01_30
Exit code: 1
Exception message: Launch container failed
[2019-08-27 11:46:07.133]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
[2019-08-27 11:46:07.134]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
{code}
My base container image only contains centos:latest, and java-1.8.0-openjdk rpm
installed. It does not have Hadoop binaries in the container. Do we need
implicit mounting of Hadoop binaries to enable existing workload to run with
runc? If not, what step can be used to run an example app?
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch,
> YARN-9561.003.patch, YARN-9561.004.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914862#comment-16914862
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 7m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
71m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 17m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}261m 43s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}397m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup |
| | hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken |
| | hadoop.yarn.server.resourcemanager.rmapp.attempt.TestAMLivelinessMonitor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978452/YARN-9561.004.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux fbf138bd5943 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0b79675 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/24617/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24617/testReport/ |
| Max. process+thread count | 4335 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24617/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> --
>
>
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914607#comment-16914607 ] Eric Badger commented on YARN-9561: --- Thanks for posting the findings, [~eyang]. I've put up patch 004 which changes the INVALID_CONFIG_FILE error into ERROR_OCI_SETUP_FAILED. Additionally, I removed the cgroupsPath requirement. For your 3rd point, I think it would be better to do this check in Java. That way we can catch the failure earlier since all containers trying to run runC will fail if overlay is not installed. For the check, I was thinking of doing an lsmod on "overlay" > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913561#comment-16913561 ] Eric Yang commented on YARN-9561: - A few discoveries found during the debug session of patch 003 with [~ebadger] and [~Jim_Brennan]: # When cluster is running without cgroup configured, container-executor will report exit code INVALID_CONFIG. This cause the node manager to become unhealthy. It would be nice to have more verbose logging for the root cause. # There are several check points are all using INVALID_CONFIG. It is hard to identify where the program failed. It would be nice to have more verbose logging for the possible root causes. # The current implementation works only with overlay module. It would be nice to detect absence of overlay module and document the requirements. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906652#comment-16906652
]
Hadoop QA commented on YARN-9561:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 13m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
60m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 13m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 13s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}255m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977521/YARN-9561.003.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 55c31dcaf6ed 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2ac029b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/24556/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24556/testReport/ |
| Max. process+thread count | 5092 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24556/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-ta
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[ https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906441#comment-16906441 ] Eric Badger commented on YARN-9561: --- Yea another error code was added, which messed up the ERROR_OCI_RUN_FAILED error code from the previous patch. Updated the patch to fix that as well as some other stuff. > Add C changes for the new RuncContainerRuntime > -- > > Key: YARN-9561 > URL: https://issues.apache.org/jira/browse/YARN-9561 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-9561.001.patch, YARN-9561.002.patch, > YARN-9561.003.patch > > > This JIRA will be used to add the C changes to the container-executor native > binary that are necessary for the new RuncContainerRuntime. There should be > no changes to existing code paths. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906360#comment-16906360
]
Eric Yang commented on YARN-9561:
-
[~ebadger] This patch no longer applies to trunk.
{code}
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci.c:
In function ‘run_oci_container’:
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci.c:849:8:
error: ‘ERROR_OCI_RUN_FAILED’ undeclared (first use in this function)
[WARNING]rc = ERROR_OCI_RUN_FAILED;
[WARNING] ^
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci.c:849:8:
note: each undeclared identifier is reported only once for each function it
appears in
[WARNING] make[2]: ***
[CMakeFiles/container.dir/main/native/container-executor/impl/oci/oci.c.o]
Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci_reap.c:
In function ‘reap_oci_layer_mounts_with_ctx’:
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci_reap.c:566:12:
error: ‘ERROR_OCI_REAP_LAYER_MOUNTS_FAILED’ undeclared (first use in this
function)
[WARNING]int rc = ERROR_OCI_REAP_LAYER_MOUNTS_FAILED;
[WARNING] ^
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci_reap.c:566:12:
note: each undeclared identifier is reported only once for each function it
appears in
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci_reap.c:
In function ‘reap_oci_layer_mounts’:
[WARNING]
/home/eyang/test/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/oci/oci_reap.c:613:12:
error: ‘ERROR_OCI_REAP_LAYER_MOUNTS_FAILED’ undeclared (first use in this
function)
[WARNING]int rc = ERROR_OCI_REAP_LAYER_MOUNTS_FAILED;
[WARNING] ^
[WARNING] make[2]: ***
[CMakeFiles/container.dir/main/native/container-executor/impl/oci/oci_reap.c.o]
Error 1
[WARNING] make[1]: *** [CMakeFiles/container.dir/all] Error 2
[WARNING] make: *** [all] Error 2
{code}
Could you check? Thanks
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9561) Add C changes for the new RuncContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-9561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16888234#comment-16888234
]
Eric Yang commented on YARN-9561:
-
[~ebadger] A few more nits while reviewing patch 002:
# In setup_container_paths, it would be good to use fprintf instead of fputs
and include the actual path location. This helps system admin to debug the
error more precisely.
# Can make_string be used instead of strbuf_append_fmt for readability reason
and reduce the need of string format functions? The 16k size seem like a limit
that is easy to reach. make_string may use more memory during string
construction, but maybe it is safer?
# This part of code doesn't seem to have any effect:
{code}
de = is_docker_support_enabled() ? enabled : disabled;
fprintf(stream,
"%11s launch docker container: %2d appid containerid workdir "
"container-script tokens pidfile nm-local-dirs nm-log-dirs "
"docker-command-file resources ", de, LAUNCH_DOCKER_CONTAINER);
{code}
I haven't looked at memory management yet, may have better way to verify when
test cases become available.
> Add C changes for the new RuncContainerRuntime
> --
>
> Key: YARN-9561
> URL: https://issues.apache.org/jira/browse/YARN-9561
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-9561.001.patch, YARN-9561.002.patch
>
>
> This JIRA will be used to add the C changes to the container-executor native
> binary that are necessary for the new RuncContainerRuntime. There should be
> no changes to existing code paths.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
