databird07 opened a new pull request, #20364: URL: https://github.com/apache/hudi/pull/20364
### Describe the issue this Pull Request addresses A GitHub CI run of `bot.yml` costs about 620 job-minutes per push (reference: run 38362333401). About 508 of those minutes are test steps and about 102 are builds. This PR collects measured changes that cut that cost without deleting test areas or losing coverage. Each change is a separate commit with a self-contained message (what, why, measurements), so any one of them can be dropped or reverted on its own. ### Summary and Changelog Every change was measured locally (JDK 17, 4 pinned CPUs, JaCoCo on): 20 runs before and 20 after, with random method order where relevant, plus JaCoCo intersection/union diffs. Every line that lost coverage was traced to another test that still covers it. | Commit | Change | Local result | Est. CI saving per push | |---|---|---|---| | `test(spark)` TestHoodieFileSystemViews | MDT-on only. Spill-to-disk writer view on every older supported version (6, 8, 9), both view types on the current version (10). Each table is written once and checked against both reader views | ~same runtime; ~750 more lines covered in every run (versions 9/10 were never tested here) | ~0 | | `test(common)` TestBootstrapIndex | 10 instead of 50 validations per thread, still 20 concurrent readers | 50.2s -> 14.5s, identical coverage | ~1 min | | `test(utilities)` DeltaStreamer matrices | Pairwise/half-fraction parameter rows, duplicate rows removed; versions 6/8/9/10 and COW/MOR kept | changed methods -29%, SchemaEvolutionQuick -21%, no class-level coverage loss | ~5 min | | `test(utilities)` KafkaTestUtils | Wait for partition leaders and check send futures (fixes a flake that dropped records and timed out after 120s, hidden by reruns) | 25 clean runs of the 22 Kafka-source tests | reliability | | `test(spark)` TestColumnStatsIndexWithSQL | Share tables across read-only cases with isolation guards. Also fixes the test never passing `tableVersion` to the writer, so v6/8/9 are really written now | -18%; always-covered lines 28,209 -> 30,174, none lost | ~2.5 min | | `chore(ci)` two surefire forks | `test-spark-java17-java-tests-part1` runs two test JVMs with per-fork ports and JaCoCo files | full part1 step 3479s -> 2078s (-40%); 4211 tests, 0 failures in every run; 60 clean subset runs | ~29 min | | `chore(ci)` build once | Three shared build jobs upload `target/` + hudi artifacts; test jobs restore instead of rebuilding | identical test counts and coverage locally; **first real GitHub Actions run is this PR** | ~30 min (projected) | **Total:** about 68 job-minutes per push (~11%). Part1, the longest Java job, drops from ~74 to ~45 minutes. Wall-clock time is still bounded by the Scala "other" job (~74 min), which Maven forking doesn't help. Not included (follow-ups): two forks on Java parts 2/3 and utilities (their profiles hard-code `forkCount=1`, and utilities needs fork-safe HiveServer2/Kafka ports), Scala job rebalancing, and Kafka-source tests that use their own producers without checking send results. ### Impact CI and test code only; no production code changes. The test matrix changes are listed per commit. The one deliberate coverage tradeoff is no Spark-record StreamSync write to a v8 table (v8 still runs with Avro records; all combinations still run on the current version). The MDT-off file-system-view cases are removed because MDT-on is the supported mode, and the MDT-off writer paths stay covered by `TestHoodieBackedMetadata` and `TestHoodieTableFileSystemView`. ### Risk Level medium. The two workflow changes are the risky part. Parallel test JVMs can expose shared-resource assumptions; this is mitigated by per-fork ports, 4 full-job runs and 60 repeated subset runs with random ordering. Build-once depends on restored outputs being treated as up to date (manifest checks on commit and profile set, re-stamped mtimes), and this PR's CI run is its first run on GitHub Actions. Both are single commits that can be dropped independently. The test-only commits are low risk. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
