[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304333#comment-17304333
]
Eric Badger commented on YARN-10495:
I would suggest using a dockerfile with the same OS version as what you plan to
run on
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303126#comment-17303126
]
angerszhu commented on YARN-10495:
--
[~ebadger] All right, we meet a problem that we use dockfile ti build hadoop
but the docker's env's glibc version is not same with our product env.
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302761#comment-17302761
]
Eric Badger commented on YARN-10495:
[~angerszhu], I don't think it's a good idea to ship glibc with Hadoop. glibc
is tied very closely to the kernel and if the ABI has changed then it won't
work.
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302262#comment-17302262
]
angerszhu commented on YARN-10495:
--
Hi [~ebadger]
When we build hadoop-3.3.0 meet glibc error
{code:java}
writev(2, [{iov_base="/usr/share/hadoop-yarn/bin/conta"..., iov_len=45},
{iov_base=": ", iov_len=2}, {iov_base="/lib64/libc.so.6", iov_len=16},
{iov_base=": ", iov_len=2}, {iov_base="version `GLIBC_2.25' not found ("...,
iov_len=75}, {iov_base="\n", iov_len=1}],
6/usr/share/hadoop-yarn/bin/container-executor: /lib64/libc.so.6: version
`GLIBC_2.25' not found (required by /lib64/x86_64/libcrypto.so.1.1)
{code}
can we also make glibc bundle to native path like `bundle.openssl` to solve
this problem?
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302168#comment-17302168
]
angerszhu commented on YARN-10495:
--
[~ebadger] Since it's a issue start from hadoop-3.3.0, so we don't need to
backport it to 3.2.0
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301569#comment-17301569
]
angerszhu commented on YARN-10495:
--
[~ebadger] Hi Eric, should we backport this pr to branch-3.3? since it's a bug
fix
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237083#comment-17237083
]
angerszhu commented on YARN-10495:
--
[~ebadger]
Seems Hadoop QA is ok now. And any else need to do?
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236584#comment-17236584
]
angerszhu commented on YARN-10495:
--
Seems test is ok.
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236401#comment-17236401
]
Hadoop QA commented on YARN-10495:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | | {color:red} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
25s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
42m 15s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
6s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
3s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 46s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
5s{color} | | {color:green} hadoop-yarn-server-nodemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | |
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236353#comment-17236353
]
Eric Badger commented on YARN-10495:
Hadoop QA seems to have failed pretty epically. Looks like OOM issues on the
box where it ran. I re-uploaded your patch (as 002) to retrigger this build
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch, YARN-10495.002.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235931#comment-17235931
]
angerszhu commented on YARN-10495:
--
[~ebadger]
Double check, this can solve our prblem.
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235890#comment-17235890
]
angerszhu commented on YARN-10495:
--
[~ebadger]
I have tested this patch in our env, and I will confirm again whether there are
other problems.
Also, I am not familiar with hadoop qa 's result, seems all UT passed?
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235658#comment-17235658
]
Hadoop QA commented on YARN-10495:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
9s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
2s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | | {color:red} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 8m
20s{color} |
[/branch-mvninstall-root.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/318/artifact/out/branch-mvninstall-root.txt]
| {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
10s{color} |
[/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/318/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt]
| {color:red} hadoop-yarn-server-nodemanager in trunk failed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} |
[/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/318/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt]
| {color:red} hadoop-yarn-server-nodemanager in trunk failed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 10m
32s{color} |
[/branch-shadedclient.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/318/artifact/out/branch-shadedclient.txt]
| {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/318/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkUbuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1.txt]
| {color:red} hadoop-yarn-server-nodemanager in trunk failed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt|https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/318/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdkPrivateBuild-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10.txt]
| {color:red} hadoop-yarn-server-nodemanager in trunk failed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
23s{color} |
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235627#comment-17235627
]
Eric Badger commented on YARN-10495:
Also, I've added you as a contributor in Hadoop Common, HDFS, Map/Reduce, and
YARN. So you will now be able to assign JIRAs to yourself (as I've already done
for you on this JIRA).
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235625#comment-17235625
]
Eric Badger commented on YARN-10495:
[~angerszhuuu], I imagine the {{-Dbundle.openssl}} adds the libcrypto.so
library to {{../lib/native}} of the build that is created? I don't have
experience with this flag. Also, have you tested this out in your environment?
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1.*
> We use a internal custom dynamic link library environment
> /usr/lib/x86_64-linux-gnu
> and we build hadoop with parameter as blow
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine shared lib library path /usr/lib/x86_64-linux-gun(where
> is libcrypto)
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager shared lib library path /usr/lib/x86_64-linux-gun(where is
> libcrypto)
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
> We build container-executor with
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10495) make the rpath of container-executor configurable
[
https://issues.apache.org/jira/browse/YARN-10495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235311#comment-17235311
]
angerszhu commented on YARN-10495:
--
ping [~ebadger] [~eyang]
> make the rpath of container-executor configurable
> -
>
> Key: YARN-10495
> URL: https://issues.apache.org/jira/browse/YARN-10495
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: angerszhu
>Priority: Major
> Attachments: YARN-10495.001.patch
>
>
> In https://issues.apache.org/jira/browse/YARN-9561 we add dependency on
> crypto to container-executor, we meet a case that in our jenkins machine, we
> have libcrypto.so.1.0.0 in shared lib env. but in our nodemanager machine we
> don't have libcrypto.so.1.0.0 but *libcrypto.so.1.1*
> we build hadoop with
> {code:java}
> -Drequire.openssl -Dbundle.openssl -Dopenssl.lib=/usr/lib/x86_64-linux-gnu
> {code}
>
> Under jenkins machine /usr/lib/x86_64-linux-gun
> {code:java}
> -rw-r--r-- 1 root root 240136 Nov 28 2014 libcroco-0.6.so.3.0.1
> -rw-r--r-- 1 root root54550 Jun 18 2017 libcrypt.a
> -rw-r--r-- 1 root root 4306444 Sep 26 2019 libcrypto.a
> lrwxrwxrwx 1 root root 18 Sep 26 2019 libcrypto.so ->
> libcrypto.so.1.0.0
> -rw-r--r-- 1 root root 2070976 Sep 26 2019 libcrypto.so.1.0.0
> lrwxrwxrwx 1 root root 35 Jun 18 2017 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 Jun 18 2017 libc.so
> {code}
>
> Under nodemanager /usr/lib/x86_64-linux-gun
> {code:java}
> -rw-r--r-- 1 root root55852 2�� 7 2019 libcrypt.a
> -rw-r--r-- 1 root root 4864244 9�� 28 2019 libcrypto.a
> lrwxrwxrwx 1 root root 16 9�� 28 2019 libcrypto.so ->
> libcrypto.so.1.1
> -rw-r--r-- 1 root root 2504576 12�� 24 2019 libcrypto.so.1.0.2
> -rw-r--r-- 1 root root 2715840 9�� 28 2019 libcrypto.so.1.1
> lrwxrwxrwx 1 root root 35 2�� 7 2019 libcrypt.so ->
> /lib/x86_64-linux-gnu/libcrypt.so.1
> -rw-r--r-- 1 root root 298 2�� 7 2019 libc.so
> {code}
>
> The libcrypto.so 's version is not same case error when we start nodemanager
>
> {code:java}
> .. 3 more Caused by:
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
> ExitCodeException exitCode=127: /home/hadoop/hadoop/bin/container-executor:
> error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared
> object file: No such file or directory at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:208)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:306)
> ... 4 more Caused by: ExitCodeException exitCode=127:
> /home/hadoop/hadoop/bin/container-executor: error while loading shared
> libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file
> or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
> ... 6 more
> {code}
>
> We should make RPATH of container-executor configurable to solve this problem
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
