codewarnab opened a new pull request, #12301: URL: https://github.com/apache/arrow-rs/pull/12301
# Which issue does this PR close? Part of #12146. # Current progress This is still in progress. The investigation and local changes cover three rounds: | Iteration | Changes investigated and added | Additional avoidable work | Total avoidable work | Reduction | |---|---|---:|---:|---:| | 1 | Cache Java/JavaScript builds; skip unnecessary disk cleanup | ~5m09s | ~5m09s | 37% | | 2 | Cache the complete Go build | ~1m16s | ~6m25s | 46% | | 3 | Cache C++ builds and reinstall native libraries | ~1m59s | ~8m24s | 60% | These calculations use the same [13m54s upstream job](https://github.com/apache/arrow-rs/actions/runs/43951890464/job/133604938577). They exclude cache overhead. The main causes were repeated peer builds, unnecessary Java modules, and disk cleanup despite ample space. ## Measured local results | Experiment | Before | After | What passed | |---|---:|---:|---| | Smaller Java build, pair 1 | 340.342s | 224.703s | 33.98% faster | | Smaller Java build, pair 2 | 333.997s | 222.488s | 33.39% faster | | C++ build versus cached restore/install | 342.215s | 1.333s | 920 build entries and 410 installed entries matched | | Rust cache experiment | 166.194s cold build | 60.123s restore + fresh-checkout rebuild | Source-change checks passed; candidate excluded pending proven CI benefit | Java averaged 33.68% faster. That gain helps cache misses; it must not be added again to warm-cache savings. Rust caching remains excluded because its restore and rebuild costs leave the CI benefit unproven. The investigation also found that GitHub already compresses caches. An additional compression layer was removed after comparison. The earlier 15% smaller-cache claim was withdrawn after testing GitHub's actual compression settings. # What changes are included in this PR? - Cache C++, Go, Java and JavaScript builds using exact source and environment keys. - Populate missing caches from main. - Build only the Java modules required by Archery. - Skip disk cleanup when at least 32 GiB is free. Rust still builds on every run. Cross-language test coverage remains unchanged. # Are these changes tested? Local artifact comparisons, integration checks and changed-value rejection checks passed: - 598 matching Java/JavaScript/Rust cases, zero failures, 139 unchanged skips. - 532 matching C++/Rust cases, zero failures, 85 unchanged skips. - Go integration checks and deliberate changed-value rejection checks passed. CI configuration checks and workflow linting also passed. The 60% figure remains projected; modified GitHub CI has not yet been measured. Real GitHub CI timings, cache hit rates, transfer costs and storage pressure still need measurement. This PR remains a draft while that work continues. # Are there any significant user-facing changes? None. # AI usage disclosure Codex helped investigate timings, implement the workflow changes and run local validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
