voonhous opened a new pull request, #19909:
URL: https://github.com/apache/hudi/pull/19909

   ### Describe the issue this Pull Request addresses
   
   Part of the CI improvement epic #19524 (#19528). Stacked on #19892. The 
commits before "ci: stop flink-2 from running hudi-flink's unit tests twice" 
belong to that PR and drop out on rebase.
   
   Three jobs repeat work that another job in the same run already does. 
Nothing consumes the second copy. This PR removes the copies and keeps one of 
each.
   
   ### Summary and Changelog
   
   Workflow files only. No test changes, no build changes, no new jobs.
   
   **1. flink-2 ran hudi-flink's unit tests by accident.**
   - Its failsafe filter was `-Dit.test=!ITTestHoodieDataSource`.
   - A filter with only an exclusion drops failsafe's `IT*` include pattern. 
Every test class matches.
   - So the job ran 170 unit-test classes, 15.5 of its 35 minutes. The docker 
job already runs them.
   - New filter: `ITTest*,!ITTestHoodieDataSource`. Checked against surefire's 
own TestListResolver. All 19 IT classes in hudi-flink are named `ITTest*`.
   
   **2. The docker job built 20 modules it never tests.**
   - Its build covers the whole reactor: 7.8 minutes. Shading 15 bundles is 4 
of them. The examples are 2 more.
   - The bundles have no tests. validate-bundles shades all 15 with the same 
profiles and then runs them.
   - The examples are on the job's own exclusion list.
   - The build now skips those 20 modules through `-pl`. Each was checked to 
exist in the reactor.
   - A fourth commit also excludes hudi-examples-k8s from the job's UT and FT 
lists. It has no tests, and it depends on the flink bundle the build no longer 
installs.
   - A fifth commit adds hudi-cli to the build skip list. It depends on the 
utilities bundle and is built and tested in test-spark-client-and-hadoop-common.
   - A sixth commit keeps the bundles out of the job's UT and FT reactors as 
well. They have no tests, and a bundle left in a test reactor tries to resolve 
modules the trimmed build no longer installs (hudi-cli-bundle failed on 
hudi-cli). The fourteen bundles are listed once as `BUNDLE_MODULES` and 
appended to all three commands; each list was checked against the reactor of 
the command it goes to, and a pom scan finds no remaining dependency from a 
kept module onto an excluded one.
   
   **3. Nine JDK 17 jobs built hudi-examples-spark.**
   - It pulls hudi-utilities, hudi-aws and hudi-gcp into the build: about 90 s 
per job.
   - Only ut-other runs the quickstart test that needs it.
   - The other eight jobs drop it. ut-other keeps it, so the module still gets 
one full build per Scala lane.
   
   ### Impact
   
   Expected against #19892's run (577 runner-minutes over 19 jobs, longest job 
flink-2 at 39):
   
   | lever | runner-minutes per run | wall clock |
   |---|---|---|
   | flink-2 filter | about -16 | flink-2 39 to about 23; the run's longest job 
becomes one of the Scala shards at about 40 |
   | docker build | about -6 | that job 36 to about 30 |
   | examples-spark | about -12 | every JDK 17 job about 1.5 min shorter |
   | together | about -34 (6%) | flink-2 leaves the critical path; the wall 
clock stays about 40, set by the Scala shards this PR does not touch |
   
   Measured against #19892's runs 
([34499656524](https://github.com/apache/hudi/actions/runs/34499656524), 
[34499656559](https://github.com/apache/hudi/actions/runs/34499656559)). Run 1 
([34617540916](https://github.com/apache/hudi/actions/runs/34617540916)) failed 
the docker job on the two dependency edges fixed in commits 5 and 6. Run 2 is 
the first fully green run; its two attempts 
([34637051799](https://github.com/apache/hudi/actions/runs/34637051799), 
[34637051750](https://github.com/apache/hudi/actions/runs/34637051750), 
attempts 1 and 2) are two samples of the same code.
   
   | | #19892 run | run 2, attempt 1 | run 2, attempt 2 | mean change |
   |---|---|---|---|---|
   | 14 Spark-lane jobs, runner-minutes | 447 | 427 | 433 | -17 |
   | their build steps, seconds summed | 5829 | 5219 | 5235 | -602 |
   | 5 engine jobs, runner-minutes | 129 | 117 | 124 | -9 |
   | all 19 jobs, runner-minutes | 577 | 544 | 558 | -26 (-5%) |
   | longest job in the run | 41 (flink-2) | 41 (scala-other-1) | 44 
(scala-other-1) | not attributable, see below |
   | flink-2 | 39, failsafe ran 187 classes | 23 | 24 | -16 |
   | docker job | 37 | 28 | 34 | -7 |
   
   The three targeted changes land where predicted: flink-2 loses 16 minutes, 
the docker job 8 to 9, the JDK 17 build steps 90 to 190 seconds each. The run's 
wall clock did not follow, because the longest job is now scala-other-1 at 41 
and 44, a job whose own code did not change in this PR and which ran 26, 42 and 
33 minutes on identical code during #19906. Its build step did get shorter 
here; its test step is the noise. So the wall-clock effect of this PR is 
"flink-2 is no longer the long pole", and what the long pole reads is set by 
the pool.
   
   <details open>
   <summary>Job by job, Spark lanes (job minutes; build step seconds, mean of 
the two attempts)</summary>
   
   | job | #19892 run | run 1 | run 2 att. 1 | run 2 att. 2 | mean change | 
build s, #19892 | build s, run 2 mean | change |
   |---|---|---|---|---|---|---|---|---|
   | test-spark-java17-ut-fat-and-ftc | 41 | 29 | 37 | 31 | -8 | 449 | 350 | 
-100 |
   | test-spark-java17-scala-other-1 | 41 | 40 | 41 | 44 | +1 | 400 | 388 | -12 
|
   | test-spark-java17-scala-dml-1 | 40 | 38 | 32 | 40 | -3 | 428 | 368 | -60 |
   | test-common-and-other-modules | 37 | failed | 28 | 34 | -7 | 601 | 402 | 
-199 |
   | test-spark-java17-ut-other | 34 | 35 | 31 | 32 | -3 | 479 | 404 | -76 |
   | test-spark-client-and-hadoop-common | 34 | 33 | 34 | 31 | -1 | 478 | 465 | 
-13 |
   | test-spark-java17-scala-other-2 | 32 | 28 | 37 | 38 | +5 | 380 | 371 | -9 |
   | test-utilities-streamer | 32 | 30 | 31 | 31 | -1 | 407 | 452 | +46 |
   | test-spark-java17-scala-dml-2 | 32 | 26 | 33 | 32 | +1 | 391 | 388 | -3 |
   | test-spark-java17-fta | 31 | 28 | 37 | 33 | +4 | 359 | 378 | +19 |
   | test-spark-java17-ut-functional | 31 | 23 | 29 | 28 | -2 | 438 | 378 | -60 
|
   | test-spark-java17-ftb | 29 | 23 | 25 | 28 | -3 | 418 | 335 | -83 |
   | test-utilities-rest | 24 | 23 | 22 | 23 | -2 | 455 | 382 | -72 |
   | test-hudi-hadoop-mr-and-hudi-java-client | 9 | 10 | 11 | 11 | +2 | 146 | 
166 | +20 |
   | total | 447 | | 427 | 433 | -17 | 5829 | 5227 | -602 |
   </details>
   
   <details open>
   <summary>Job by job, engines (minutes)</summary>
   
   | job | #19892 run | run 1 | run 2 att. 1 | run 2 att. 2 | mean change |
   |---|---|---|---|---|---|
   | test-flink-2 | 39 | 23 | 23 | 24 | -16 |
   | integration-tests | 33 | 38 | 37 | 37 | +4 |
   | test-flink-1 | 28 | 32 | 31 | 32 | +3 |
   | validate-bundles | 19 | 19 | 15 | 20 | -2 |
   | validate-bundle-spark4 | 10 | 11 | 11 | 12 | +1 |
   | total | 129 | 123 | 117 | 124 | -9 |
   </details>
   
   <details open>
   <summary>Every module the docker job stops building, and where the same step 
still runs</summary>
   
   Counts are from these job logs: 
[validate-bundles](https://github.com/apache/hudi/actions/runs/34608288732/job/103292067057),
 [the docker 
job](https://github.com/apache/hudi/actions/runs/34590618744/job/103234823835) 
before this PR, 
[ut-other](https://github.com/apache/hudi/actions/runs/34590618744/job/103234823609),
 
[flink-1](https://github.com/apache/hudi/actions/runs/34602459140/job/103273046201).
 The flink-2 duplicate unit-test run is visible in 
[flink-2](https://github.com/apache/hudi/actions/runs/34602459140/job/103273046252):
 187 classes under failsafe, 170 of them not named ITTest.
   
   | module no longer built by the docker job | before this PR | after this PR |
   |---|---|---|
   | packaging/hudi-hadoop-mr-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-datahub-sync-bundle | shaded twice: docker job build (1x) 
and validate-bundles (1x) | shaded in validate-bundles (1x), then run by its 
bundle validation |
   | packaging/hudi-hive-sync-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-aws-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-gcp-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-azure-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-spark-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-presto-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-utilities-slim-bundle | shaded twice: docker job build (1x) 
and validate-bundles (1x) | shaded in validate-bundles (1x), then run by its 
bundle validation |
   | packaging/hudi-timeline-server-bundle | shaded twice: docker job build 
(1x) and validate-bundles (1x) | shaded in validate-bundles (1x), then run by 
its bundle validation |
   | packaging/hudi-flink-bundle | shaded twice: docker job build (1x) and 
validate-bundles (2x) | shaded in validate-bundles (2x), then run by its bundle 
validation |
   | packaging/hudi-kafka-connect-bundle | shaded twice: docker job build (1x) 
and validate-bundles (1x) | shaded in validate-bundles (1x), then run by its 
bundle validation |
   | packaging/hudi-cli-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | packaging/hudi-metaserver-server-bundle | shaded twice: docker job build 
(1x) and validate-bundles (1x) | shaded in validate-bundles (1x), then run by 
its bundle validation |
   | packaging/hudi-utilities-bundle | shaded twice: docker job build (1x) and 
validate-bundles (1x) | shaded in validate-bundles (1x), then run by its bundle 
validation |
   | hudi-examples/hudi-examples-common | compiled in the docker job build (1x) 
and in validate-bundles (1x) | compiled in validate-bundles |
   | hudi-examples/hudi-examples-spark | compiled in the docker job build (1x) 
and in validate-bundles (1x) | compiled in validate-bundles, and ut-other which 
also runs its quickstart test |
   | hudi-examples/hudi-examples-flink | compiled in the docker job build (1x) 
and in validate-bundles (1x) | compiled in validate-bundles, and flink-1 which 
also runs its unit tests |
   | hudi-examples/hudi-examples-java | compiled in the docker job build (1x) 
and in validate-bundles (1x) | compiled in validate-bundles |
   | hudi-examples/hudi-examples-k8s | compiled in the docker job build (1x) 
and in validate-bundles (1x) | compiled in validate-bundles |
   
   validate-bundles passes `-DdeployArtifacts=true`, which activates the 
`hudi-platform-service` profile, so it shades the metaserver server bundle and 
the metaserver client inside the spark and flink bundles exactly as the docker 
job did.
   </details>
   
   Coverage is unchanged: every module still goes through its full build, 
checkstyle, scalastyle and RAT included, at least once per Scala lane in the 
same run, every bundle is still shaded once in every profile shape CI produces, 
and every unit test still runs once. What goes away is the second, third and 
ninth copy of the same result.
   
   ### Risk Level
   
   low
   
   Workflow files only. If the flink-2 filter matched too little, the job would 
report fewer integration tests than before; the count to compare is 18 IT 
classes. A wrong module path in the docker exclusion list fails that build 
immediately, which the first run shows.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to