voonhous commented on code in PR #19884:
URL: https://github.com/apache/hudi/pull/19884#discussion_r3975192963
##########
.github/workflows/bot.yml:
##########
@@ -1507,9 +1507,28 @@ jobs:
SPARK_ARCHIVE: ${{ matrix.sparkArchive }}
SCALA_PROFILE: '-Dscala-2.12 -Dscala.binary.version=2.12'
run: |
- echo "Downloading $SPARK_ARCHIVE"
- curl --retry 5 https://archive.apache.org/dist/spark/$SPARK_ARCHIVE
--create-dirs -o $GITHUB_WORKSPACE/$SPARK_ARCHIVE
- tar -xvf $GITHUB_WORKSPACE/$SPARK_ARCHIVE -C $GITHUB_WORKSPACE/
+ # dlcdn only carries the current release of each line; fall back to
+ # the archive for older pins (#19883). Plain --retry, not
+ # --retry-all-errors, so a 404 on the CDN falls through immediately.
+ DEST="$GITHUB_WORKSPACE/$SPARK_ARCHIVE"
+ downloaded=false
+ for base in https://dlcdn.apache.org/spark
https://archive.apache.org/dist/spark; do
+ echo "Downloading $SPARK_ARCHIVE from $base"
+ if curl -fL --create-dirs -o "$DEST" \
+ --retry 5 --retry-delay 10 \
+ --connect-timeout 30 --speed-limit 100000 --speed-time 120 \
+ -C - "$base/$SPARK_ARCHIVE"; then
Review Comment:
Not intended, good catch. Fixed in 314a97b.
You are right that `--retry` restarts from zero, and `-C -` was worse than a
no-op on retries here: the offset is computed once at startup, and the file
never exists at that point, because the CDN attempt starts from nothing and the
loop `rm -f`s the partial before falling back to the archive. There was no
reachable path where it could resume anything. Confirmed against a throttled
local server -- three attempts, all sending `Range=None`, and the file left
holding one attempt's bytes rather than three. Dropped it.
The floor was the actual bug. `100000` is not a stall threshold, it is the
archive's bad-day throughput: the 98 KB/s run in #19883 is 100,352 B/s (curl
prints K as 1024), and the abort fires on any 120s window, so that run would
have burned six restarts and exited 1 where the old command finished slowly.
Lowered to `1000`, which only a dead connection sits under. Same throttled
server at 32KB/s aborts with exit 28 at `--speed-limit 100000` and completes at
`1000`; one that sends headers and then nothing still aborts at `1000`. There
is a comment in the workflow recording why the floor has to sit well below the
archive's real speed.
I left resume out rather than moving `--retry` into the shell loop. Doing
that properly also wants `.sha512` verification so a spliced or truncated
tarball fails before `tar` rather than during it (both bases serve the
sidecar), and that is more shell than this PR needs -- the low floor plus `-f`
covers the failure modes it set out to fix. Happy to add it if you would rather
the archive path be genuinely resumable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
