Jeffwan commented on issue #27347: [SPARK-30626][K8S] Add SPARK_APPLICATION_ID into driver pod env URL: https://github.com/apache/spark/pull/27347#issuecomment-578059691 @dongjoon-hyun BTW, when I run ` ./dev/dev-run-integration-tests.sh` I notice a problem. Based container image doesn't have package cache, and `apt install` encounter problems ``` Step 6/12 : RUN apt install -y python python-pip && apt install -y python3 python3-pip && rm -r /usr/lib/python*/ensurepip && pip install --upgrade pip setuptools && rm -r /root/.cache && rm -rf /var/cache/apt/* ---> Running in 74256170f862 WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Reading package lists... Building dependency tree... Reading state information... The following additional packages will be installed: ........ python2-minimal python2.7 python2.7-dev python2.7-minimal readline-common shared-mime-info xdg-user-dirs xz-utils Suggested packages: ..... python2.7-doc binfmt-support readline-doc The following NEW packages will be installed: .... python2.7-minimal readline-common shared-mime-info xdg-user-dirs xz-utils 0 upgraded, 130 newly installed, 0 to remove and 0 not upgraded. Need to get 112 MB of archives. After this operation, 367 MB of additional disk space will be used. Get:1 http://deb.debian.org/debian buster/main amd64 perl-modules-5.28 all 5.28.1-6 [2873 kB] Get:3 http://deb.debian.org/debian buster/main amd64 libgdbm6 amd64 1.18.1-4 [64.7 kB] Get:4 http://deb.debian.org/debian buster/main amd64 libgdbm-compat4 amd64 1.18.1-4 [44.1 kB] ..... Get:128 http://deb.debian.org/debian buster/main amd64 python-xdg all 0.25-5 [35.9 kB] Get:129 http://deb.debian.org/debian buster/main amd64 shared-mime-info amd64 1.10-1 [766 kB] Get:130 http://deb.debian.org/debian buster/main amd64 xdg-user-dirs amd64 0.17-2 [53.8 kB] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7-minimal_2.7.16-2_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/python2.7-minimal_2.7.16-2_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7-stdlib_2.7.16-2_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/python2.7_2.7.16-2_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://security-cdn.debian.org/debian-security/pool/updates/main/l/linux/linux-libc-dev_4.19.67-2+deb10u1_amd64.deb 404 Not Found [IP: 151.101.52.204 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/g/glib2.0/libglib2.0-0_2.58.3-2+deb10u1_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/g/glib2.0/libglib2.0-data_2.58.3-2+deb10u1_all.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7_2.7.16-2_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/libpython2.7-dev_2.7.16-2_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python2.7/python2.7-dev_2.7.16-2_amd64.deb 404 Not Found [IP: 151.101.54.133 80] E: Failed to fetch http://deb.debian.org/debian/pool/main/p/python-cryptography/python-cryptography_2.6.1-3_amd64.deb 404 Not Found [IP: 151.101# .54.133 80] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? Fetched 71.1 MB in 6s (12.3 MB/s) The command '/bin/sh -c apt install -y python python-pip && apt install -y python3 python3-pip && rm -r /usr/lib/python*/ensurepip && pip install --upgrade pip setuptools && rm -r /root/.cache && rm -rf /var/cache/apt/*' returned a non-zero code: 100 Failed to build PySpark Docker image, please refer to Docker build output for details. ``` Some paths may not be available and paths needs to be updated, I add `apt update` before `apt install` in the Python and R docker files. Do you think it's worth to file a PR to mitigate this issue?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
