voonhous commented on code in PR #19884:
URL: https://github.com/apache/hudi/pull/19884#discussion_r3975192963
##########
.github/workflows/bot.yml:
##########
@@ -1507,9 +1507,28 @@ jobs:
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run: |
- echo "Downloading $SPARK_ARCHIVE"
- curl --retry 5 https://archive.apache.org/dist/spark/$SPARK_ARCHIVE
--create-dirs -o $GITHUB_WORKSPACE/$SPARK_ARCHIVE
- tar -xvf $GITHUB_WORKSPACE/$SPARK_ARCHIVE -C $GITHUB_WORKSPACE/
+ # dlcdn only carries the current release of each line; fall back to
+ # the archive for older pins (#19883). Plain --retry, not
+ # --retry-all-errors, so a 404 on the CDN falls through immediately.
+ DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ downloaded=false
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 100000 --speed-time 120 \
+ -C - "$base/$SPARK_ARCHIVE"; then
Review Comment:
Not intended, good catch. Fixed in 314a97b.
Removed `-C` - because this retry flow never resumes a partial download.
Also lowered `--speed-limit` from `100000` to `1000` to avoid repeatedly
aborting slow but healthy archive downloads.
Verified with a throttled local server that slow downloads complete and
stalled connections still time out. Left resumable downloads and checksum
verification out of scope for this PR.
With `--speed-limit 1000 --speed-time 120`, curl aborts if the transfer
stays **below 1,000 bytes/s for 120 seconds**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
