voonhous opened a new pull request, #19884:
URL: https://github.com/apache/hudi/pull/19884
### Describe the issue this Pull Request addresses
Part of #19883. The `integration-tests` job downloads a 382MB Spark tarball
from `archive.apache.org` on every run. That host is a single un-CDN'd origin
and was measured between **42 seconds and 1h 08m** for the same file, which at
worst turned a 37-minute job into 106 minutes.
Separately, `pom.xml` compiled against 3.5.5 while the job ran its Spark
tests on a **3.5.3** runtime. That is the unsafe direction: code built against
3.5.5 can reference APIs absent from 3.5.3.
### Summary and Changelog
Both halves have the same root cause, so they are fixed together.
- `spark35.version` 3.5.5 -> **3.5.9**, and the `integration-tests` archive
pin moves to match. 3.5.9 is the current 3.5 release, so it is served by
`dlcdn.apache.org` at CDN speed instead of by the archive. The 5
`spark-avro_2.12:3.5.5` spark-submit examples move with it, since spark-avro
has to match the runtime it is submitted against.
- The download tries `dlcdn.apache.org` first and falls back to
`archive.apache.org`, because the CDN carries only the current release of each
line and any pin eventually rolls off it.
- `curl` gains `-f` so an error page becomes a download failure rather than
a corrupt tarball, and `--speed-limit`/`--speed-time` so a transfer stuck under
100KB/s for two minutes is aborted and resumed on a fresh connection. `tar`
loses `-v`, which was printing 4000 lines of filenames into the job log.
<details>
<summary>Why plain --retry and not --retry-all-errors</summary>
`--retry-all-errors` retries a 404, so a dlcdn miss would sleep through five
pointless retries before falling back -- roughly 50 seconds on exactly the path
that matters. Plain `--retry` still covers transient 5xx and stalls but returns
immediately on 404. Measured 0.06s vs 6.09s on a real 404. There is a comment
in the workflow recording this so it does not get "fixed" back.
</details>
<details>
<summary>Measured download times behind the 42s to 68min range</summary>
From curl's own summary line in each job log, same URL, same file, same
runner label:
| Download | Avg speed | Job wall clock | Run |
|---|---|---|---|
| 0:00:42 | 9.0 MB/s | 37m | 34314808833 |
| 0:05:52 | 1.1 MB/s | 42m | 34306895783 |
| 0:11:56 | 546 KB/s | 51m | 34237534196 |
| 0:35:04 | 186 KB/s | 73m | 34228343560 |
| 0:42:06 | 154 KB/s | 78m | 34226281866 |
| 0:48:32 | 134 KB/s | 85m | 34226858343 |
| 1:07:42 | 98 KB/s | 106m | 34316284096 |
The `mvn verify` that follows in the same step is constant at 21.8 to 24.1
minutes across all of these, so the variance is the download alone. Full logs
are attached to #19883.
</details>
<details>
<summary>What the tarball is for, and what is deliberately not in this
PR</summary>
The downloaded distribution is the runtime for `hudi-cli`'s eight
`ITTest*Command` classes, which use `SparkLauncher` to resolve
`$SPARK_HOME/bin/spark-submit` (`SparkUtil.java:53`). The docker-cluster ITs in
`hudi-integ-test` are all `@Disabled` (HUDI-8274, HUDI-8440) and report `Tests
run: 9, Skipped: 9`, so there is no client-versus-cluster version coupling to
break here.
Three Spark 3.5 pins are still stale and are **not** touched by this PR,
because they need new base images published to the `apachehudi` Docker Hub org
first (`packaging/bundle-validation/Dockerfile` does `FROM
apachehudi/hudi-ci-bundle-validation-base:$IMAGE_TAG`):
- bundle-validation lanes at `spark3.5.0` and `spark3.5.1`
- the hive-sync job's `sparkadhoc_3.5.3` image
- the `docker/compose` spark353 stack
Tracked on #19883.
</details>
### Impact
CI only. No production code paths change. The `integration-tests` job should
drop from a 42s-to-68min download to a few seconds, removing roughly 11 minutes
of avoidable runner time per run against the ASF 4,200 h/week cap tracked in
#19524.
### Risk Level
**low**. CI-only change. The three Spark sources Hudi copies from are
byte-identical between `v3.5.5` and `v3.5.9`, and every in-tree Spark version
gate is minor-level, so the patch bump crosses none of them.
<details>
<summary>Verification detail</summary>
Byte-identical between `v3.5.5` and `v3.5.9`: `ParquetFileFormat.scala`,
`TypeCoercion.scala`, `DecimalPrecision.scala` (compared by git blob SHA). The
parquet one is identical from `v3.5.1`, the revision `Spark35ParquetReader`
cites. Version gates in tree are all minor-level: `startsWith("3.5")`, `>=
"3.5.0"`.
- JDK 17 `mvn install` of `hudi-spark3.5.x_2.12 -am`: BUILD SUCCESS across
16 modules including every Spark-internal shim, resolving
`spark-core_2.12-3.5.9`.
- JDK 11 `TestSavepointRestoreCopyOnWrite` under `-Pfunctional-tests`: 7/7
pass with `Running Spark version 3.5.9` in the log.
- Download loop exercised on all three paths: CDN hit (0s), CDN 404 falling
back to archive (1s), absent everywhere failing with exit 1 and a clear message
(1s).
Scala 2.13 was not gated locally; 3.5.9 artifacts are confirmed on Maven
Central for 2.13 and CI covers that lane.
</details>
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
