zaynt4606 commented on code in PR #3877:
URL: https://github.com/apache/celeborn/pull/3877#discussion_r4311079873
##########
build/sbt-launch-lib.bash:
##########
@@ -73,16 +79,25 @@ acquire_sbt_jar () {
# Download
printf "Attempting to fetch sbt\n"
JAR_DL="${JAR}.part"
- if [ $(command -v curl) ]; then
- curl --fail --location --silent ${URL1} > "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- elif [ $(command -v wget) ]; then
- wget --quiet ${URL1} -O "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- else
- printf "You do not have curl or wget installed, please install sbt
manually from https://www.scala-sbt.org/\n";
- exit -1
+ CURL_RETRY_OPTS="--retry 3 --retry-delay 5"
+ # --retry alone skips DNS, connection and 403 errors; --retry-all-errors
needs curl 7.71+.
+ if [ $(command -v curl) ] && curl --help all 2>/dev/null | grep -q --
"--retry-all-errors"; then
+ CURL_RETRY_OPTS="${CURL_RETRY_OPTS} --retry-all-errors"
fi
+ for URL in ${URLS}; do
+ if [ $(command -v curl) ]; then
+ curl --fail --location --silent --show-error ${CURL_RETRY_OPTS} ${URL}
> "${JAR_DL}" &&\
Review Comment:
`--retry-all-errors` can retry after a transfer has written a partial JAR.
With `> "${JAR_DL}"`, the shell keeps that partial output across curl's
retries, so a later successful attempt can append to it. Curl may then exit
successfully and the following `mv` installs a corrupt JAR, bypassing the
mirror fallback. The [curl
manual](https://curl.se/docs/manpage.html#--retry-all-errors) explicitly warns
against redirected output here. Could you use `--output "${JAR_DL}"` instead so
curl can reset the file between attempts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
