voonhous opened a new pull request, #19919: URL: https://github.com/apache/hudi/pull/19919
### Describe the issue this Pull Request addresses Part of the CI improvement epic #19524 (#19527), tracked on #19910. Stacked on #19918; the commits before "ci: build and verify only the IT modules in integration-tests" belong to that stack and drop out on rebase. The integration-tests job builds all 66 modules of the reactor (8.9 min), then runs `mvn verify` over the same 66 modules, which packages every one of them a second time, the 17 bundles included, before running the integration tests (21 min). The integration tests live in five modules. ### Summary and Changelog - `IT_MODULES` names the five modules with integration-test classes outside hudi-flink, whose ITs run in the flink jobs: hudi-cli, hudi-integ-test, hudi-aws, hudi-azure, and hudi-gcp-bundle (its `ITGcpBundleCompatibility` came with #20020). The docker demo suites in hudi-integ-test are skipped and `@Disabled`, and the job has no docker-compose activity. - The build step passes them with `-am`. Maven resolves that to 31 modules: the core chain through hudi-utilities, the utilities bundle for hudi-cli, the hadoop-mr, hive-sync and spark bundles plus two docker pom modules for hudi-integ-test, and hudi-gcp-bundle itself (hudi-gcp is already in the chain). - The IT step verifies just the five, instead of the whole reactor minus hudi-flink. Every module no longer built here is built, and where a bundle shaded and run, in the job that owns it: | no longer built in integration-tests | already built and tested or validated in | |---|---| | hudi-flink, hudi-flink-client, the Flink datasource modules | test-flink-1 and test-flink-2 | | hudi-flink-bundle and the other ten bundles the ITs do not need | validate-bundles | | hudi-examples-common, java, k8s | validate-bundles | | hudi-examples-spark | ut-other and validate-bundles | | hudi-examples-flink | test-flink-1 and validate-bundles | | hudi-kafka-connect | test-common-and-other-modules (tests), validate-bundles (bundle) | | hudi-datahub-sync, hudi-adb-sync | test-common-and-other-modules | | metaserver modules | test-common-and-other-modules, validate-bundles | | the verify step's second packaging of all 66 modules | this job's own build step | What stays: the 31-module build and all 15 integration-test classes the job runs today (8 in hudi-cli, 2 in hudi-aws, 1 in hudi-azure, 1 in hudi-gcp-bundle, hudi-integ-test's skipped ones). ### Impact Expected against #19892's run ([34499656559](https://github.com/apache/hudi/actions/runs/34499656559)): build step 8.9 to about 5 minutes, verify step 21 to about 16, the job from 33-38 minutes to the high twenties, about 10 runner-minutes per run. The job sits on the plateau with the Scala shards, so the wall clock follows only if those come down too. Measured on [34670498928](https://github.com/apache/hudi/actions/runs/34670498928), four attempts, against #19918's run [34668461644](https://github.com/apache/hudi/actions/runs/34668461644), which has the same code minus this change: | integration-tests | #19918 run | this PR (attempts 1 to 4) | |---|---|---| | Build Project step (s) | 646 | 479, 456, 515, 495 | | IT step (s) | 1390 | 1028, 971, 985, 1006 | | modules built / verified | 66 / 64 | 30 / 4 | | job execution (min) | 37 | 28, 25, 27, 28 | The IT step runs the same 25 test classes in both logs (the 14 integration-test classes plus hudi-integ-test's 11 unit classes), with the same per-module totals. The run's 19 test jobs total 539, 531, 551 and 516 runner-minutes over the four attempts (mean 534), against 541 for #19918's run. The other 18 jobs each drift by up to 10 minutes between attempts with no code change (attempt 3: fta and scala-dml-2 both +10; attempt 4: utilities-streamer -10), so a single total cannot resolve a 10-minute change in one job; the job's own steps are the measurement. Across about 90 earlier samples of this job on master and PRs since 2026-09-04, the IT step never ran under 1258 s. Longest job unchanged at 40 to 41 min. Per-job tables in the comments below. These runs predate the rebase onto #20020, which adds hudi-gcp-bundle to both steps (one more module built and verified, one more IT class). ### Risk Level low Both `-pl` lists validate against the job's profiles, and `-am` makes the build's reactor exactly the dependency closure of what verify runs, so nothing verify needs can be missing. If a module gains integration tests, `IT_MODULES` is the one line to extend (hudi-gcp-bundle was added here when #20020 gave it an integration test); the failsafe class count to compare after this PR is 15. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
