This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c1a03b2 [SPARK-31200][K8S] Enforce to use `https` in
/etc/apt/sources.list
c1a03b2 is described below
commit c1a03b2233aee77f957dbc94180d2334c61ac088
Author: Prashant Sharma <[email protected]>
AuthorDate: Fri Mar 27 09:13:55 2020 -0700
[SPARK-31200][K8S] Enforce to use `https` in /etc/apt/sources.list
…n progress errors.
### What changes were proposed in this pull request?
Switching to `https` instead of `http` in the debian mirror urls.
### Why are the changes needed?
My ISP was trying to intercept (or trying to serve from cache) the `http`
traffic and this was causing a very confusing errors while building the spark
image. I thought by posting this, I can help someone save his time and energy,
if he encounters the same issue.
```
bash-3.2$ bin/docker-image-tool.sh -r scrapcodes -t v3.1.0-f1cc86 build
Sending build context to Docker daemon 203.4MB
Step 1/18 : ARG java_image_tag=8-jre-slim
Step 2/18 : FROM openjdk:${java_image_tag}
---> 381b20190cf7
Step 3/18 : ARG spark_uid=185
---> Using cache
---> 65c06f86753c
Step 4/18 : RUN set -ex && apt-get update && ln -s /lib /lib64 &&
apt install -y bash tini libc6 libpam-modules krb5-user libnss3 procps &&
mkdir -p /opt/spark && mkdir -p /opt/spark/examples && mkdir -p
/opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln
-sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >>
/etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&
rm -rf /var/cache/apt/*
---> Running in 96bcbe927d35
+ apt-get update
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Err:3 http://deb.debian.org/debian buster/main amd64 Packages
File has unexpected size (13217 != 7906744). Mirror sync in progress?
[IP: 151.101.10.133 80]
Hashes of expected file:
- Filesize:7906744 [weak]
- SHA256:80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc
- MD5Sum:80b6d9c1b6630b2234161e42f4040ab3 [weak]
Release file created at: Sat, 08 Feb 2020 10:57:10 +0000
Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages [7380
B]
Err:5 http://deb.debian.org/debian buster-updates/main amd64 Packages
File has unexpected size (13233 != 7380). Mirror sync in progress? [IP:
151.101.10.133 80]
Hashes of expected file:
- Filesize:7380 [weak]
- SHA256:6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d
Release file created at: Fri, 20 Mar 2020 02:28:11 +0000
Get:4 http://security-cdn.debian.org/debian-security buster/updates
InRelease [65.4 kB]
Get:6 http://security-cdn.debian.org/debian-security buster/updates/main
amd64 Packages [183 kB]
Fetched 419 kB in 1s (327 kB/s)
Reading package lists...
E: Failed to fetch
http://deb.debian.org/debian/dists/buster/main/binary-amd64/by-hash/SHA256/80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc
File has unexpected size (13217 != 7906744). Mirror sync in progress? [IP:
151.101.10.133 80]
Hashes of expected file:
- Filesize:7906744 [weak]
-
SHA256:80ed5d1cc1f31a568b77e4fadfd9e01fa4d65e951243fd2ce29eee14d4b532cc
- MD5Sum:80b6d9c1b6630b2234161e42f4040ab3 [weak]
Release file created at: Sat, 08 Feb 2020 10:57:10 +0000
E: Failed to fetch
http://deb.debian.org/debian/dists/buster-updates/main/binary-amd64/by-hash/SHA256/6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d
File has unexpected size (13233 != 7380). Mirror sync in progress? [IP:
151.101.10.133 80]
Hashes of expected file:
- Filesize:7380 [weak]
-
SHA256:6af9ea081b6a3da33cfaf76a81978517f65d38e45230089a5612e56f2b6b789d
Release file created at: Fri, 20 Mar 2020 02:28:11 +0000
E: Some index files failed to download. They have been ignored, or old ones
used instead.
The command '/bin/sh -c set -ex && apt-get update && ln -s /lib
/lib64 && apt install -y bash tini libc6 libpam-modules krb5-user libnss3
procps && mkdir -p /opt/spark && mkdir -p /opt/spark/examples &&
mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh
&& ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so
use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw
/etc/passwd && rm -rf /var/cache/apt [...]
Failed to build Spark JVM Docker image, please refer to Docker build output
for details.
```
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Manually by switching to `https` mirrors on the offending ISP (I am already
on).
Closes #27966 from ScrapCodes/docker-mirror.
Authored-by: Prashant Sharma <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit f87957371dc9c723013849c65b014c0fbf93ed3a)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile | 1 +
1 file changed, 1 insertion(+)
diff --git
a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
index 6ed37fc..c1df215 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile
@@ -28,6 +28,7 @@ ARG spark_uid=185
# docker build -t spark:latest -f kubernetes/dockerfiles/spark/Dockerfile .
RUN set -ex && \
+ sed -i 's/http:/https:/g' /etc/apt/sources.list && \
apt-get update && \
ln -s /lib /lib64 && \
apt install -y bash tini libc6 libpam-modules krb5-user libnss3 && \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
