DanielLeens opened a new pull request, #11639:
URL: https://github.com/apache/seatunnel/pull/11639
### Purpose of this pull request
Follow-up to #10799, which added the `DebeziumAdapter` SPI and per-connector
`debezium.version` properties as a staged foundation for per-connector Debezium
version management.
Those properties do not yet control the Debezium a connector runs on.
`connector-cdc-base` declares `debezium-api` and `debezium-embedded` at
`compile` scope, so `debezium-core` reaches every CDC connector transitively at
the version resolved in **`connector-cdc-base`'s own POM**. Setting
`<debezium.version>` in `connector-cdc-mysql` changes only
`debezium-connector-mysql`; the Debezium core stays pinned by the shared module.
That is not just a POM detail. `AbstractPluginDiscovery#filterPluginJar`
special-cases CDC and forces `connector-cdc-base` into the class loader of
**every** CDC plugin, requiring exactly two jars:
```java
if (pluginName.contains("cdc")) {
return pathname.getName().endsWith(".jar")
&& (StringUtils.startsWithIgnoreCase(pathname.getName(),
pluginJarPrefix)
|| StringUtils.startsWithIgnoreCase(pathname.getName(),
"connector-cdc-base"));
}
```
Checking the published jars confirms where Debezium actually lives today:
| jar | `io.debezium` core/embedded classes | `debezium-connector-*` classes
|
|---|---|---|
| `connector-cdc-base` | **294** | 0 |
| `connector-cdc-mysql` | **0** | 158 |
So the shared `connector-cdc-base` jar is the single project-wide Debezium
version, and no connector can override it. This PR moves that ownership to the
connectors.
### Does this PR introduce _any_ user-facing change?
No change to CDC job configuration, runtime behavior or checkpoint
compatibility. Every in-tree connector still resolves Debezium `1.9.8.Final`,
so the set of classes on a CDC job's class loader is unchanged — only relocated
from the shared jar into the connector jar.
Two things do change for people building against SeaTunnel, both documented
in `docs/{en,zh}/introduction/concepts/incompatible-changes.md`:
- `connectors/connector-cdc-base-*.jar` no longer contains `io.debezium`
classes.
- Third-party CDC connectors that relied on `connector-cdc-base` to supply
Debezium transitively must now declare `debezium-api` and `debezium-embedded`
themselves.
### What changed
**Dependency ownership**
- `connector-cdc-base` declares `debezium-api`, `debezium-embedded` and
`zstd-jni` as `provided`. It still compiles against Debezium but no longer
imposes a version on its consumers, and its shaded jar is Debezium-free.
- The Debezium dependency management moves up to the `connector-cdc` parent.
Inherited dependency management is interpolated against the effective
properties of the module being built, so each connector resolves Debezium at
**its own** `debezium.version` while sharing one copy of the exclusion rules
(`kafka-log4j-appender`, glassfish jersey, the Apple-M1 `zstd-jni` override).
- Every Debezium-based connector declares `debezium-api`,
`debezium-embedded` and `zstd-jni` explicitly. `connector-cdc-opengauss` and
`connector-cdc-vitess` gain the `debezium.version` property they were missing
in #10799.
- `connector-cdc-tidb` is intentionally untouched — it uses no Debezium
classes. Verified at the bytecode level: the only `connector-cdc-base` classes
it touches (`SourceOptions`, `StartupMode`, `TemporalConversions`) contain zero
`io/debezium` or `org/apache/kafka` constant-pool references, because
`CompatibleDebeziumJsonDeserializationSchema.IDENTIFIER` is a compile-time
constant that javac inlines.
**The SPI gets real providers**
Each connector now registers a `DebeziumAdapter` via `META-INF/services`, so
the extension point added in #10799 is populated and the version a connector
ships is introspectable at runtime.
`connector-cdc-opengauss` deliberately registers none. It runs Debezium's
PostgreSQL connector and shades `connector-cdc-postgres` into its own jar, and
the shade plugin is configured with `ServicesResourceTransformer`, which
**merges** `META-INF/services` entries. A second provider for
`io.debezium.connector.postgresql.PostgresConnector` would therefore land in
the same jar and make every openGauss CDC job fail the exactly-one-match rule.
`OpengaussDebeziumAdapterTest` pins that so a future contributor cannot
reintroduce it silently.
### How was this patch tested?
Unit tests per connector assert that the adapter resolves uniquely for its
connector class, that it claims no other connector class, and — the drift guard
— that its declared version equals the `debezium-core` version **actually
resolved onto the module classpath** (read from
`META-INF/maven/io.debezium/debezium-core/pom.properties`). A POM bump that
forgets the adapter, or vice versa, fails the build.
The ownership claim itself was verified with `dependency:tree`. Before,
`debezium-core` was fixed for everyone. After, with only
`connector-cdc-mysql`'s property temporarily set to `1.9.7.Final`:
```
connector-cdc-base
io.debezium:debezium-api:jar:1.9.8.Final:provided
io.debezium:debezium-embedded:jar:1.9.8.Final:provided
io.debezium:debezium-core:jar:1.9.8.Final:provided
connector-cdc-mysql
io.debezium:debezium-api:jar:1.9.7.Final:compile
io.debezium:debezium-embedded:jar:1.9.7.Final:compile
io.debezium:debezium-core:jar:1.9.7.Final:compile
io.debezium:debezium-connector-mysql:jar:1.9.7.Final:compile
connector-cdc-postgres
io.debezium:debezium-api:jar:1.9.8.Final:compile
io.debezium:debezium-embedded:jar:1.9.8.Final:compile
io.debezium:debezium-core:jar:1.9.8.Final:compile
```
Two CDC connectors resolving genuinely different Debezium cores — not
possible before this change. That probe was reverted; everything in this PR is
on `1.9.8.Final`.
### Trade-off for reviewers
**This increases the size of the binary distribution.** The Debezium runtime
is no longer shared: `connector-cdc-base` drops from ~24 MB to roughly 1 MB
(only ~0.37 MB of it was ever SeaTunnel code — the rest is Kafka Connect ~4.9
MB, Guava, Jetty, snappy, jackson and the `zstd-jni` native binaries), and each
of the 7 Debezium-based connector jars gains that payload instead. Net growth
is on the order of +140 MB.
That cost is inherent to the goal: if a connector can pin its own Debezium,
it has to carry it. Sharing one copy is exactly what makes overriding
impossible today.
If the size is not acceptable, the alternative worth discussing is shipping
**versioned shared Debezium runtime jars** (e.g.
`connector-cdc-debezium-1.9.8`) and having `AbstractPluginDiscovery` load the
one each connector declares. That reclaims the sharing for connectors on the
same version while still allowing divergence, but it rewires the
plugin-discovery path — a runtime change deliberately kept out of this staged
PR. Raising it here as a draft so that direction can be settled before anything
in the runtime path moves.
### Check list
* [x] Code changed are covered with tests
* [x] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
— no new dependencies are introduced; existing Debezium artifacts are
re-scoped only
* [x] If necessary, please update the documentation to describe the new
feature — `incompatible-changes.md` updated in `en` and `zh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
