wombatu-kun commented on code in PR #19884:
URL: https://github.com/apache/hudi/pull/19884#discussion_r3974967633
##########
.github/workflows/bot.yml:
##########
@@ -1507,9 +1507,28 @@ jobs:
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run: |
- echo "Downloading $SPARK_ARCHIVE"
- curl --retry 5 https://archive.apache.org/dist/spark/$SPARK_ARCHIVE
--create-dirs -o $GITHUB_WORKSPACE/$SPARK_ARCHIVE
- tar -xvf $GITHUB_WORKSPACE/$SPARK_ARCHIVE -C $GITHUB_WORKSPACE/
+ # dlcdn only carries the current release of each line; fall back to
+ # the archive for older pins (#19883). Plain --retry, not
+ # --retry-all-errors, so a 404 on the CDN falls through immediately.
+ DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ downloaded=false
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 100000 --speed-time 120 \
+ -C - "$base/$SPARK_ARCHIVE"; then
Review Comment:
`-C -` never resumes here: the offset is computed once from a file that does
not exist yet, and curl truncates back to it before every `--retry`, so a
`--speed-time` abort restarts all 382MB from zero. The archive averages in
#19883 sit just above the 100KB/s floor, so that path can now burn five
restarts and `exit 1` where it used to finish slowly - intended, or should the
retry move into the shell loop so `-C -` can resume?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
