Jiajia Li created SPARK-28347:
---------------------------------
Summary: Error loading glibc library when generatting data on HDFS
in K8S
Key: SPARK-28347
URL: https://issues.apache.org/jira/browse/SPARK-28347
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.4.3, 3.0.0
Reporter: Jiajia Li
I want to run TPC-DS in K8S, when generating data on HDFS, with following
error:
{code:java}
ERROR Utils: Aborting task
java.lang.UnsatisfiedLinkError:
/tmp/snappy-1.1.7-91d7ec9a-6e2d-47af-90ae-84c00d143b6d-libsnappyjava.so: Error
loading shared library ld-linux-x86-64.so.2: No such file or directory
{code}
Looking into the executor pod, ld-linux-x86-64.so.2 is existed in /lib64 but
not in /lib :
{code:java}
# ls -al /lib/ld*
-rwxr-xr-x 1 root root 584304 Mar 19 09:56 /lib/ld-musl-x86_64.so.1
# ls -al /lib64/ld* lrwxrwxrwx 1 root root 26 Jul 2 09:05
/lib64/ld-linux-x86-64.so.2 -> /lib/libc.musl-x86_64.so.1
{code}
After adding the gcompat in Dockerfile, ld-linux-x86-64.so.2 exists in both
lib and lib64 folders and no more error:
{code:java}
ls -al /lib/ld*
-rwxr-xr-x 1 root root 18512 Dec 24 2018 /lib/ld-linux-x86-64.so.2
-rwxr-xr-x 1 root root 584304 Mar 19 09:56 /lib/ld-musl-x86_64.so.1
ls -al /lib64/ld*
lrwxrwxrwx 1 root root 26 Jul 11 02:06 /lib64/ld-linux-x86-64.so.2 ->
/lib/libc.musl-x86_64.so.1
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
