goutamadwant opened a new pull request, #12475: URL: https://github.com/apache/seatunnel/pull/12475
### Purpose of this pull request Related: #11513 (PR validation redesign). This is a small, independent routing change; it does not overlap #11515 (docs/UI/dependency routing and impacted unit tests) or #9976 (dedicated jobs for long-running modules). For connector-only PRs, `backend.yml` starts all 8 `updated-modules-integration-test-part-N` jobs on JDK 8 and 11 whenever `it-modules` is non-empty. The modules are split round-robin (`i % 8`), so a PR that touches one or two connectors fills one or two shards. The other jobs still check out the repo, set up Java, free disk space, and then print `sub modules is empty, skipping`. Example: fork run https://github.com/goutamadwant/seatunnel/actions/runs/35958959241 (Cassandra connector change, `it-modules = ["connector-cassandra-e2e", "connector-cassandra-e2e"]`): - 16 shard legs started. Only `part-1` (JDK 8 and 11) ran a test: 33 runner-minutes. - The other 14 legs each took 1.1 to 3.2 minutes and ran no test: 29.9 runner-minutes and 14 extra runner slots. They started between 06:26Z and 07:59Z, so they also queued for runners behind the fork's concurrent-job limit. This PR: - adds `build_sub_update_it_shards()` / `sub_update_it_shards` to `tools/update_modules_check/update_modules_check.py`. It returns the shard names (`part-1` … `part-8`) that get at least one module. It reuses the same exclusion and round-robin logic that each shard job already uses, so the shard list and the per-shard module list can't drift apart. - adds a `Make updated integration test shards` step and a `updated-it-shards` output to the `changes` job. - adds `&& contains(fromJSON(needs.changes.outputs.updated-it-shards), 'part-N')` to each `updated-modules-integration-test-part-N` condition. The existing conditions and the in-job "empty, skipping" guard stay as they are. Risk controls: - If the helper fails, the step logs a warning and outputs all 8 shards. So a script error runs everything, as today, and never skips everything. - The per-shard module assignment does not change. A module runs in the same shard as before. - A new unit test reads `backend.yml` and checks that each `part-N` job is gated on `'part-N'` and still calls `sub_update_it_module "$IT_MODULES" 8 <N-1>`, and that the `changes` step uses the same shard count (8). - api/engine PRs are unaffected: those jobs were already skipped for them. ### Does this PR introduce _any_ user-facing change? No. CI only. ### How was this patch tested? 1. Unit tests (the same command the `Code style` job runs): ``` python3 -m unittest discover -s tools/update_modules_check -p 'test_*.py' Ran 33 tests ... OK ``` New tests: - one module → `["part-1"]` - three modules → `part-1..3` - only dedicated-job modules (Kafka, k8s, engine smoke) → `[]` - 11 modules → all 8 shards - shard list agrees with `sub_update_it_module` output for every index - a non-positive shard count is rejected - workflow gating matches shard indexes Before the workflow change, the new workflow test failed (`changes job must compute the non-empty updated-modules shards`). After the change, it passes. 2. YAML parsing of `backend.yml` (PyYAML); `git diff --check`. 3. A local routing simulation script (`simulate_routing.py`). It uses the filter globs from `backend.yml`, `ci_scope.py` for `api`, and the module lists from the example runs. It evaluates every job's `if:` and expands matrices. | Changed files | Job legs started, before → after | Shard legs with no module, before → after | |---|---|---| | One connector (Cassandra, as run 35958959241) | 26 → 12 | 14 → 0 | | Three connectors (http, mongodb, jdbc) | 26 → 20 | 6 → 0 | | Kafka only (its e2e module has a dedicated job) | 28 → 12 | 16 → 0 | | Docs only | 7 → 7 | 0 → 0 | | Engine UI only (`engine=true`) | 79 → 79 | 0 → 0 | | Transform only (`api=true`) | 83 → 83 | 0 → 0 | The simulation's "before" count for the Cassandra case, 26 legs plus the docs job that the real PR also triggered, matches the 27 jobs in the real run. Fork CI on this branch: https://github.com/goutamadwant/seatunnel/actions/runs/36140951336. It was queued when this PR was opened. This change touches only workflow and CI-tooling files, so no updated-modules shards run on it. The gating is shown by the unit tests and the routing simulation above. On the first connector-only PR after merge, `changes` should print the shard list, for example `["part-1"]`, and the other `updated-modules-integration-test-part-N` legs should show as skipped. Not verified: `actionlint` was not run locally. ### Check list * [ ] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md) * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR. * [ ] If you are contributing the connector code, please check that the following files are updated: 1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it 2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) 3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) 4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/) 5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
