david-streamlio opened a new pull request, #92:
URL: https://github.com/apache/pulsar-connectors/pull/92
### Motivation
The `Tests - Connectors` job runs every module's tests on a single runner.
As container-backed suites landed today (Debezium ×7, Kafka, Solr,
DynamoDB/LocalStack, InfluxDB v1+v2, MariaDB, ClickHouse, elastic-search, ...),
its wall-clock roughly **doubled — from ≈8–10 min to ≈20 min**. Gradle
parallelizes within the job, but `ubuntu-latest` gives only 2 cores, so there
is little headroom left to exploit in-process.
### Modifications
Split the single test job into **three matrix shards** that run on separate
runners in parallel:
| Shard | Modules |
|---|---|
| **Debezium** | the 7 `debezium` modules — Oracle's ~1.6 GB image and
LogMiner startup dominate |
| **Streaming & Search** | `elastic-search` (8 Testcontainers classes),
`kafka`, `kafka-connect-adaptor`, `kinesis`, `dynamodb` |
| **Other** | catch-all: `test integrationTest` for **every remaining
module**, via `-x` exclusions of the two explicit groups |
The **Other** shard is a catch-all by design: a newly added module is picked
up by it automatically and can never be silently dropped. The only maintenance
rule — stated as an `INVARIANT` comment in the file — is that Other's `-x` list
must mirror the two explicit groups exactly.
### Verifying this change
The shards are a **disjoint cover** of the full test set — verified with
`--dry-run`, which is authoritative for what each command resolves to:
```
ALL = 40 test/integrationTest tasks
Debezium = 7, Streaming & Search = 5, Other = 28 → 7 + 5 + 28 = 40
in ALL but no shard (would be SKIPPED): (empty)
pairwise overlaps G1∩G2, G1∩G3, G2∩G3: (all empty)
```
So every task runs in exactly one shard; nothing is skipped or double-run.
YAML validated (`YAML.load_file`); the `>-` folded scalars collapse to the
single-line command `./gradlew ${{ matrix.tasks }}` expects.
### Expected effect
Wall-clock for the test stage drops from one ~20-min job to roughly
`max(shard)` + the fixed ~1–2 min setup, since the shards run concurrently.
Each shard also gets its own 2 cores, so Gradle parallelism improves within a
shard too.
### Notes
- Balance is approximate — Debezium (Oracle) and Streaming & Search
(elastic-search) are the two heaviest clusters, deliberately separated. If one
shard dominates in practice, modules can be moved between groups with no
structural change.
- Complementary to #91, which bounds the MQTT container startup so a stalled
pull fails fast instead of consuming a shard's 45-minute budget.
- Retains `fail-fast: false`, so one shard failing still lets the others
report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
