voonhous opened a new issue, #19883: URL: https://github.com/apache/hudi/issues/19883
**What needs to be done:** Stop the `integration-tests` job from pulling `spark-3.5.3-bin-hadoop3.tgz` (382 MB) off `archive.apache.org` on every run. `bot.yml:1268`: ```bash curl --retry 5 https://archive.apache.org/dist/spark/$SPARK_ARCHIVE --create-dirs -o $GITHUB_WORKSPACE/$SPARK_ARCHIVE ``` **Why this task is needed:** Same URL, same file, same `ubuntu-latest` label, measured over the last day: **42 seconds to 1h 08m**. A 97x spread, taken from curl's own summary line in each job log. | Download | Avg speed | Job wall clock | Run | |---|---|---|---| | 0:00:42 | 9.0 MB/s | 37m | [34314808833](https://github.com/apache/hudi/actions/runs/34314808833/job/102351735379) | | 0:05:52 | 1.1 MB/s | 42m | [34306895783](https://github.com/apache/hudi/actions/runs/34306895783/job/102331391385) | | 0:11:56 | 546 KB/s | 51m | [34237534196](https://github.com/apache/hudi/actions/runs/34237534196/job/102142162860) | | 0:35:04 | 186 KB/s | 73m | [34228343560](https://github.com/apache/hudi/actions/runs/34228343560/job/102088674900) | | 0:42:06 | 154 KB/s | 78m | [34226281866](https://github.com/apache/hudi/actions/runs/34226281866/job/102061097243) | | 0:48:32 | 134 KB/s | 85m | [34226858343](https://github.com/apache/hudi/actions/runs/34226858343/job/102080340820) | | 1:07:42 | 98 KB/s | 106m | [34316284096](https://github.com/apache/hudi/actions/runs/34316284096/job/102438976959) | The `mvn verify` that follows in the same step is constant at 21.8 to 24.1 minutes across every one of these runs. All of the variance is the download. <details> <summary>Raw log lines behind the table (Actions logs age out after 90 days)</summary> Three lines per run: the step's own `Downloading` echo, curl's final summary, and the first line `tar` emits. The gap between the first and third timestamp is the download. ``` run 34226281866 job 102061097243 2026-09-08T13:35:48.5819876Z Downloading spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz 2026-09-08T14:17:55.3808194Z 100 382M 100 382M 0 0 154k 0 0:42:06 0:42:06 --:--:-- 214k 2026-09-08T14:17:55.3891626Z spark-3.5.3-bin-hadoop3/ run 34226858343 job 102080340820 2026-09-08T15:03:30.8229922Z Downloading spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz 2026-09-08T15:52:03.5695988Z 100 382M 100 382M 0 0 134k 0 0:48:32 0:48:32 --:--:-- 308k 2026-09-08T15:52:03.5765032Z spark-3.5.3-bin-hadoop3/ run 34228343560 job 102088674900 2026-09-08T15:27:31.9684020Z Downloading spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz 2026-09-08T16:02:36.4096889Z 100 382M 100 382M 0 0 186k 0 0:35:04 0:35:04 --:--:-- 129k 2026-09-08T16:02:36.4173602Z spark-3.5.3-bin-hadoop3/ run 34237534196 job 102142162860 2026-09-08T18:34:27.0075332Z Downloading spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz 2026-09-08T18:46:23.2263115Z 100 382M 100 382M 0 0 546k 0 0:11:56 0:11:56 --:--:-- 596k 2026-09-08T18:46:23.2340156Z spark-3.5.3-bin-hadoop3/ run 34306895783 job 102331391385 2026-09-09T04:09:51.6423462Z Downloading spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz 2026-09-09T04:15:44.6263112Z 100 382M 100 382M 0 0 1109k 0 0:05:52 0:05:52 --:--:-- 7043k 2026-09-09T04:15:44.6328269Z spark-3.5.3-bin-hadoop3/ run 34314808833 job 102351735379 2026-09-09T05:57:16.3431886Z Downloading spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz 2026-09-09T05:57:58.6616140Z 100 382M 100 382M 0 0 9252k 0 0:00:42 0:00:42 --:--:-- 4289k 2026-09-09T05:57:58.6697430Z spark-3.5.3-bin-hadoop3/ run 34316284096 job 102438976959 2026-09-09T09:09:20.5434168Z Downloading spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz 2026-09-09T10:17:03.2337497Z 100 382M 100 382M 0 0 98670 0 1:07:42 1:07:42 --:--:-- 170k 2026-09-09T10:17:03.2398071Z spark-3.5.3-bin-hadoop3/ ``` </details> Three things follow: 1. **Runner budget.** 8 of the 20 recent `integration-tests` runs sampled paid more than 2 minutes, averaging about 11 minutes of avoidable runner time per run. That is charged against the 4,200 h/week ASF cap tracked in #19524. 2. **Wall clock.** The worst case turned a 37-minute job into 106 minutes, and it is the long pole for the whole workflow when it hits. 3. **Timeout risk.** At the 98 KB/s worst case we are 68 minutes into a 6-hour job limit. A further 3x degradation on the archive fails the job outright, and the current `curl` has no `-f`, so a 5xx body gets written into the `.tgz` and surfaces as a confusing `tar` error rather than a download failure. **There is no faster mirror to switch to.** `archive.apache.org` is a single un-CDN'd origin. The Fastly-backed CDN only carries the current release per line, and 3.5.3 rolled off it: | Source | Has 3.5.3 | Measured | |---|---|---| | `archive.apache.org` | yes | 66 KB/s | | `dlcdn.apache.org` | no, 404 (carries 3.5.8, 3.5.9) | 70 MB/s on 3.5.9 | | `downloads.apache.org` | no, 404 | -- | | tuna, ustc, aliyun, jaist, aarnet | no, 404 | -- | | `mirrors.huaweicloud.com` | yes | 225 KB/s | So the version pin is what puts us on the slow path, and any fix has to address that rather than swap hostnames. <details> <summary>Options, none picked yet</summary> 1. **Cache the tarball** (`actions/cache`, keyed on the archive path). Only a cold run touches the archive; PR runs restore from the default branch cache at GitHub-internal speed. 382 MB against the 10 GB repo quota. Fixes it regardless of which version we pin, and composes with #19529. 2. **Track a version that is still on the CDN.** Puts us at ~6 s, but rolls off again in a few months, so it needs a fallback chain and periodic bumps. 3. **Pull from the `apache/spark` Docker images.** Every patch tag from 3.5.3 through 3.5.9 is published; `docker cp` out of `apache/spark:3.5.3-scala2.12-java11-ubuntu` keeps the exact version and is fast from Actions. Subject to anonymous Docker Hub pull limits. 4. **Harden the `curl` regardless of source.** Worth doing on its own: ```bash curl -fL --create-dirs -o "$DEST" \ --retry 5 --retry-all-errors --retry-delay 10 \ --connect-timeout 30 --speed-limit 100000 --speed-time 120 \ -C - "$url" ``` `--speed-limit`/`--speed-time` abort a transfer stuck under 100 KB/s for two minutes so the retry can land on a different archive backend; `-C -` resumes rather than restarting. `-f` turns an error page into a download failure instead of a corrupt archive. </details> <details> <summary>Two adjacent findings</summary> - `pom.xml:189` sets `spark35.version` to `3.5.5`, but `bot.yml:1227` downloads `3.5.3`. The integration tests run against an older runtime than the code is compiled against, and nothing pins the two together. - The same `archive.apache.org` dependency exists in 8 Dockerfiles under `docker/` and `packaging/bundle-validation/` (hadoop, hive, derby, flink, spark, thrift tarballs). Those are baked into published images so they do not cost per-run, but they inherit the same availability risk when an image is rebuilt. </details> **Task Type:** Performance optimization **Parent feature issue:** #19524 (Wall clock stream) **Related issues:** #19529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
