tkobayas commented on PR #6860: URL: https://github.com/apache/incubator-kie/pull/6860#issuecomment-5489141998
## PR update: Partition files approach Thanks for the review @tiagobento. I've reworked the PR based on your feedback. Here's a summary of the changes. ### Label-based switching The parallel CI workflow (`ci-parallel.yaml`) is activated by attaching the **`parallel ci`** label to a PR. Without the label, the original sequential `ci.yaml` runs as before. This lets us opt in per-PR and makes it easy to revert if Apache Infra raises concerns about GHA runner quota — each partition job rebuilds upstream modules independently, so total runner-minutes increase even though wall-clock time decreases. Keeping both `ci.yaml` and `ci-parallel.yaml` would be a maintenance burden. We may finally combine them, but keep them separated for a while as `ci.yaml` is the stable CI and `ci-parallel.yaml` is the opt-in experimental CI. WDYT? ### Partition files Replaced the pom.xml `BEGIN/END` markers with partition files in `.github/supporting-files/ci/partitions/`: - `partition0.txt` — optaplanner (14 leaf modules) - `partition1.txt` — kogito-runtimes (87 leaf modules) - `partition2.txt` — kogito-apps (39 leaf modules) - Implicit default — drools + shared modules (no file needed) Adding or removing a partition is just adding or removing a file. The GHA workflow uses a dynamic matrix (`compute-partitions` job discovers partition files and outputs a JSON array, `test` job uses `partition × java × os` cross-product), so no YAML changes are needed. **Exclusive assignment**: each affected module goes to a partition only if it's in exactly one partition's transitive closure. Shared modules (like `kie-api`, `kogito-api`) go to the implicit default. This avoids redundant tests — each module is tested in exactly one job. The current 4 partitions are just a starting point. We can split further (e.g., `kogito-data-index`, `kogito-jobs-service`) by adding more partition files. ### 2-phase upstream build (image-producer modules) The upstream build uses `-T 1C` for parallelism, but Quarkus runtime extensions have an implicit dependency on their `-deployment` counterpart that Maven doesn't capture. To avoid race conditions: **Phase 1** — Build all upstream modules in parallel with Quarkus disabled: ``` -T 1C -Dquarkus.build.skip=true -Ddisable.quarkus.plugin=true -Dskip.quarkus.image.assembly=true ``` **Phase 2** — Rebuild image-producing modules normally (without the skip flags). These modules need Quarkus augmentation to produce `target/quarkus-app` and container image assemblies required by integration tests. The module list is maintained in `image-producers.txt`. This 2-phase approach would also benefit `ci.yaml`, which currently uses `-T 1C` in its upstream build and has the same latent Quarkus race condition. The five image-producing modules that use `maven-assembly-plugin` to package `target/quarkus-app` require a small POM change: a `skip.quarkus.image.assembly` property to skip the assembly when Quarkus is disabled (otherwise the archive fails as empty). ### What this PR does NOT do - **`-T 1C` for the test phase** — parallel test execution with surefire/failsafe is risky (port conflicts, flaky tests). This could be explored in a separate PR. ### CI results All checks green (run `89788266903`). Wall-clock time comparison (all modules affected): | Job | Java 17 ||| Java 21 ||| |---|---:|---:|---:|---:|---:|---:| | | upstream | test | **total** | upstream | test | **total** | | compute-partitions | — | — | **3m 06s** | — | — | — | | partition0 (optaplanner) | 1m 52s | 9m 42s | **14m 18s** | 1m 50s | 9m 13s | **13m 36s** | | partition1 (kogito-runtimes) | 5m 09s | 42m 09s | **51m 56s** | 6m 21s | 49m 36s | **58m 50s** | | partition2 (kogito-apps) | 8m 23s | 1h 03m | **1h 14m 30s** | 7m 04s | 1h 18m | **1h 29m 02s** | | default (drools + shared) | 5m 29s | 52m 50s | **1h 00m 33s** | 5m 35s | 54m 54s | **1h 02m 50s** | Wall-clock: **~1h 32m** (vs ~3h sequential). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
