voonhous opened a new pull request, #19919: URL: https://github.com/apache/hudi/pull/19919
### Describe the issue this Pull Request addresses Part of the CI improvement epic #19524 (#19527), tracked on #19910. Stacked on #19918; the commits before "ci: build and verify only the IT modules in integration-tests" belong to that stack and drop out on rebase. The integration-tests job builds all 66 modules of the reactor (8.9 min), then runs `mvn verify` over the same 66 modules, which packages every one of them a second time, the 17 bundles included, before running the integration tests (21 min). The integration tests live in four modules. ### Summary and Changelog - `IT_MODULES` names the four modules with integration-test classes outside hudi-flink, whose ITs run in the flink jobs: hudi-cli, hudi-integ-test, hudi-aws, hudi-azure. The docker demo suites in hudi-integ-test are skipped and `@Disabled`, and the job has no docker-compose activity. - The build step passes them with `-am`. Maven resolves that to 30 modules: the core chain through hudi-utilities, the utilities bundle for hudi-cli, and the hadoop-mr, hive-sync and spark bundles plus two docker pom modules for hudi-integ-test. - The IT step verifies just the four, instead of the whole reactor minus hudi-flink. Every module no longer built here is built, and where a bundle shaded and run, in the job that owns it: | no longer built in integration-tests | already built and tested or validated in | |---|---| | hudi-flink, hudi-flink-client, the Flink datasource modules | test-flink-1 and test-flink-2 | | hudi-flink-bundle and the other eleven bundles the ITs do not need | validate-bundles | | hudi-examples-common, java, k8s | validate-bundles | | hudi-examples-spark | ut-other and validate-bundles | | hudi-examples-flink | test-flink-1 and validate-bundles | | hudi-kafka-connect | test-common-and-other-modules (tests), validate-bundles (bundle) | | hudi-datahub-sync, hudi-adb-sync | test-common-and-other-modules | | metaserver modules | test-common-and-other-modules, validate-bundles | | the verify step's second packaging of all 66 modules | this job's own build step | What stays: the 30-module build and all 14 integration-test classes the job runs today (8 in hudi-cli, 2 in hudi-aws, 1 in hudi-azure, hudi-integ-test's skipped ones). ### Impact Expected against #19892's run ([34499656559](https://github.com/apache/hudi/actions/runs/34499656559)): build step 8.9 to about 5 minutes, verify step 21 to about 16, the job from 33-38 minutes to the high twenties, about 10 runner-minutes per run. The job sits on the plateau with the Scala shards, so the wall clock follows only if those come down too. Measured numbers go here after the run. ### Risk Level low Both `-pl` lists validate against the job's profiles, and `-am` makes the build's reactor exactly the dependency closure of what verify runs, so nothing verify needs can be missing. If a module gained integration tests, `IT_MODULES` is the one line to extend; the failsafe class count to compare after this PR is 14. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
