This is an automated email from the ASF dual-hosted git repository.
david-streamlio pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-connectors.git
The following commit(s) were added to refs/heads/master by this push:
new 28ef0cc1 [improve][ci] Shard the test job across three parallel
runners (#92)
28ef0cc1 is described below
commit 28ef0cc1603119ca51d85b288d57ef06a07decf4
Author: David Kjerrumgaard <[email protected]>
AuthorDate: Fri Jul 10 09:26:16 2026 -0700
[improve][ci] Shard the test job across three parallel runners (#92)
The Tests - Connectors job ran every module's tests on one runner. As
container-backed suites landed (Debezium, Kafka, Solr, DynamoDB,
InfluxDB, MariaDB, ...) its wall-clock roughly doubled to ~20 minutes,
and Gradle's in-job parallelism is capped at the runner's 2 cores.
Split it into three matrix shards that run on separate runners:
- Debezium: the seven debezium modules (Oracle's 1.6GB image dominates)
- Streaming & Search: elastic-search, kafka, kafka-connect-adaptor,
kinesis, dynamodb
- Other: a catch-all running 'test integrationTest' for every remaining
module, via -x exclusions of the two explicit groups
The catch-all guarantees a newly added module is always covered by
exactly one shard. Verified with --dry-run that the three shards form a
disjoint cover of all 40 test/integrationTest tasks: 7 + 5 + 28 = 40,
no task skipped, no overlap.
---
.github/workflows/ci.yaml | 40 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 38 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
index 3bf4adb0..7f5f0354 100644
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -69,9 +69,45 @@ jobs:
strategy:
fail-fast: false
matrix:
+ # The test job is sharded across module groups so container-backed
suites run on
+ # separate runners in parallel. "Other" is a catch-all: it runs `test
integrationTest`
+ # for every module EXCEPT those claimed by the explicit groups, so a
newly added module
+ # is always covered by exactly one shard and can never be silently
skipped.
+ #
+ # INVARIANT: the `-x` exclusions in "Other" must list exactly the
modules named in the
+ # "Debezium" and "Streaming & Search" groups — no more, no fewer.
include:
- - name: Connectors
- tasks: test integrationTest
+ - name: Debezium
+ tasks: >-
+ :debezium:pulsar-io-debezium-core:test
+ :debezium:pulsar-io-debezium-mariadb:test
+ :debezium:pulsar-io-debezium-mongodb:test
+ :debezium:pulsar-io-debezium-mssql:test
+ :debezium:pulsar-io-debezium-mysql:test
+ :debezium:pulsar-io-debezium-oracle:test
+ :debezium:pulsar-io-debezium-postgres:test
+ - name: Streaming & Search
+ tasks: >-
+ :elastic-search:test
+ :kafka:test
+ :kafka-connect-adaptor:test
+ :kinesis:test
+ :dynamodb:test
+ - name: Other
+ tasks: >-
+ test integrationTest
+ -x :debezium:pulsar-io-debezium-core:test
+ -x :debezium:pulsar-io-debezium-mariadb:test
+ -x :debezium:pulsar-io-debezium-mongodb:test
+ -x :debezium:pulsar-io-debezium-mssql:test
+ -x :debezium:pulsar-io-debezium-mysql:test
+ -x :debezium:pulsar-io-debezium-oracle:test
+ -x :debezium:pulsar-io-debezium-postgres:test
+ -x :elastic-search:test
+ -x :kafka:test
+ -x :kafka-connect-adaptor:test
+ -x :kinesis:test
+ -x :dynamodb:test
steps:
- uses: actions/checkout@v6
