This is an automated email from the ASF dual-hosted git repository.
zaynt4606 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new eca67458ed [CELEBORN-2484][BUILD] Make sbt downloads resilient to
transient Maven Central failures
eca67458ed is described below
commit eca67458ed3a3b9a92ba24a675890a78628b7024
Author: afterincomparableyum
<[email protected]>
AuthorDate: Mon Oct 19 17:15:04 2026 +0800
[CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven
Central failures
### What changes were proposed in this pull request?
- `build/sbt-launch-lib.bash`: the sbt launcher download now retries. It
adds `--retry-all-errors` when curl is 7.71 or newer, because plain `--retry`
skips DNS, connection and 403 errors. It prints the download error, and if
repo1 still fails it falls back to Google's Maven Central mirror. That fallback
is skipped when `DEFAULT_ARTIFACT_REPOSITORY` is set.
- `build/sbt-config/repositories`: point `gcs-maven-central-mirror` at the
live `/maven2/` path. The old `/repos/central/data/` path returns 404.
`repositories-asia.template` already uses `/maven2/`.
### Why are the changes needed?
SBT CI jobs fail before any test runs whenever Maven Central has a brief
error:
- The launcher was fetched with a single silent `curl` and no retry. The
job died in under a second with only `Our attempt to download sbt locally ...
failed` in the log ([job
1](https://github.com/apache/celeborn/actions/runs/38999585335/job/117299481319?pr=3875),
[job
2](https://github.com/apache/celeborn/actions/runs/38999734574/job/117299925875?pr=3876)).
The Actions cache is at its 10GB limit, so the cached launcher is usually
evicted and every matrix job downloads it.
- The GCS mirror path was dead, so every dependency came from repo1 alone.
A single connection reset failed the build ([job
3](https://github.com/apache/celeborn/actions/runs/38376216131/job/115225130981):
`could not retrieve sbt 1.9.4`).
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [ ] Yes
### How was this patch tested?
- Ran the launcher download locally three ways: normal, with repo1
returning 404, and with repo1 failing DNS. Each case retried as expected, fell
back to the mirror and produced a valid jar.
- `curl` against the GCS mirror: `/maven2/` returns 200 and
`/repos/central/data/` returns 404.
- CI on this PR covers the full dependency resolution through the corrected
mirror.
Closes #3877 from afterincomparableyum/sbt-launcher-download-retry.
Authored-by: afterincomparableyum
<[email protected]>
Signed-off-by: zhengtao <[email protected]>
---
build/sbt-config/repositories | 2 +-
build/sbt-launch-lib.bash | 35 +++++++++++++++++++++++++----------
2 files changed, 26 insertions(+), 11 deletions(-)
diff --git a/build/sbt-config/repositories b/build/sbt-config/repositories
index 628c5ea553..ea9de31395 100644
--- a/build/sbt-config/repositories
+++ b/build/sbt-config/repositories
@@ -24,7 +24,7 @@
# fetched from the environment variable `DEFAULT_ARTIFACT_REPOSITORY` and
# assigned within the build/sbt-launch-lib.bash script.
private: ${celeborn.sbt.default.artifact.repository-file:///dev/null}
- gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/repos/central/data/
+ gcs-maven-central-mirror:
https://maven-central.storage-download.googleapis.com/maven2/
maven-central
typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/,
[organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext],
bootOnly
sbt-ivy-snapshots: https://repo.scala-sbt.org/scalasbt/ivy-snapshots/,
[organization]/[module]/[revision]/[type]s/[artifact](-[classifier]).[ext],
bootOnly
diff --git a/build/sbt-launch-lib.bash b/build/sbt-launch-lib.bash
index 858bb2f80c..d9fba0cca3 100755
--- a/build/sbt-launch-lib.bash
+++ b/build/sbt-launch-lib.bash
@@ -62,7 +62,13 @@ acquire_sbt_jar () {
if [ -n "$DEFAULT_ARTIFACT_REPOSITORY" ]; then
export SBT_OPTS="$SBT_OPTS
-Dceleborn.sbt.default.artifact.repository=${DEFAULT_ARTIFACT_REPOSITORY}"
fi
-
URL1=${DEFAULT_ARTIFACT_REPOSITORY:-https://repo1.maven.org/maven2/}org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
+
SBT_LAUNCH_PATH=org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
+ if [ -n "$DEFAULT_ARTIFACT_REPOSITORY" ]; then
+ URLS="${DEFAULT_ARTIFACT_REPOSITORY}${SBT_LAUNCH_PATH}"
+ else
+ # Fall back to Google's Maven Central mirror when repo1 is unreachable or
rate limited.
+ URLS="https://repo1.maven.org/maven2/${SBT_LAUNCH_PATH}
https://maven-central.storage-download.googleapis.com/maven2/${SBT_LAUNCH_PATH}";
+ fi
JAR=build/sbt-launch-${SBT_VERSION}.jar
sbt_jar=$JAR
@@ -73,16 +79,25 @@ acquire_sbt_jar () {
# Download
printf "Attempting to fetch sbt\n"
JAR_DL="${JAR}.part"
- if [ $(command -v curl) ]; then
- curl --fail --location --silent ${URL1} > "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- elif [ $(command -v wget) ]; then
- wget --quiet ${URL1} -O "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- else
- printf "You do not have curl or wget installed, please install sbt
manually from https://www.scala-sbt.org/\n";
- exit -1
+ CURL_RETRY_OPTS="--retry 3 --retry-delay 5"
+ # --retry alone skips DNS, connection and 403 errors; --retry-all-errors
needs curl 7.71+.
+ if [ $(command -v curl) ] && curl --help all 2>/dev/null | grep -q --
"--retry-all-errors"; then
+ CURL_RETRY_OPTS="${CURL_RETRY_OPTS} --retry-all-errors"
fi
+ for URL in ${URLS}; do
+ if [ $(command -v curl) ]; then
+ curl --fail --location --silent --show-error ${CURL_RETRY_OPTS}
--output "${JAR_DL}" ${URL} &&\
+ mv "${JAR_DL}" "${JAR}"
+ elif [ $(command -v wget) ]; then
+ wget --quiet --tries=4 --waitretry=5 ${URL} -O "${JAR_DL}" &&\
+ mv "${JAR_DL}" "${JAR}"
+ else
+ printf "You do not have curl or wget installed, please install sbt
manually from https://www.scala-sbt.org/\n";
+ exit -1
+ fi
+ [ -f "${JAR}" ] && break
+ printf "Failed to fetch sbt from ${URL}\n"
+ done
fi
if [ ! -f "${JAR}" ]; then
# We failed to download
