tkaymak opened a new pull request, #39569: URL: https://github.com/apache/beam/pull/39569
Addresses #39568. Upgrades `DebeziumIO` from Debezium `3.1.3.Final` to `3.5.2.Final`. `3.6.0` is deliberately skipped: it shipped a rewritten MySQL DDL parser with several open regressions and there is no `3.6.1` yet. A follow-up bump should be trivial once that line settles. ### Commits The branch is split so the mechanical bump can be reviewed separately from the two behavioural pieces: 1. **`Remove dead snapshot.mode override in DebeziumIO.getRecordSchema`** — `getRecordSchema` copied the connector config, set `snapshot.mode` on the copy, then passed the *original* map to `getOneRecord`. The override never took effect. The dead copy is dropped rather than honoured, because `getOneRecord` samples a real data record to derive the schema, so a schema-only snapshot would emit nothing and inference would fail. No behaviour change, and this commit stands alone against master. 2. **`Upgrade DebeziumIO to Debezium 3.5.2.Final`** — the upgrade proper. 3. **`Wire DebeziumSDFDatabaseHistory default to schema.history.internal`** — a genuine behaviour fix, kept separate so it can be dropped independently if reviewers prefer. ### Notable points for review * **Kafka Connect 4.1.2.** Debezium 3.5 builds against Connect 4.1.2, so `connect-api` moves 3.9.0 → 4.1.2 with `kafka-clients` declared and forced to match. Connect 4.1 added an abstract `pluginMetrics()` to `SourceTaskContext`; `BeamSourceTaskContext` returns `null`. I verified no Debezium 3.5.2 connector jar references `pluginMetrics`, so this is unreachable in practice. * **`debezium-core` module split.** 3.5 split core into smaller modules, so `debezium-config` and `debezium-connector-common` are declared explicitly. I checked the expansion-service shadow jar still bundles `io/debezium/config/Configuration.class`, `AbstractSchemaHistory`, and all five connectors. * **Postgres JDBC driver pin.** The 3.5 Postgres connector needs driver 42.7.x. `42.7.7` is forced **in this module only**, matching the upstream Debezium 3.5.2 pin. The Beam-wide `42.6.2` CVE pin from 24f562d84f6 is untouched. * **Third commit is a real behaviour change.** The default schema history impl was registered under `database.history`, the pre-Debezium-2.0 key that 3.x ignores. `DebeziumSDFDatabaseHistory` was therefore unreachable by default, and historized connectors silently fell back to `KafkaSchemaHistory` pointing at `localhost:9092`. Users who set `schema.history.internal` themselves are unaffected. ### Test changes Two adaptations to 3.5 behaviour, not masked failures: * Connection failures now surface wrapped in a `RetriableException` rather than a top-level `DebeziumException`, so the schema transform tests unwrap the cause chain. * The `example-mysql` image sets `log_bin = mysql-bin`, so the binlog is `mysql-bin.000003` rather than `binlog.000002`. I confirmed this is deterministic for a fresh container rather than a rotation artifact. ### Validation Run locally on JDK 17: * `:sdks:java:io:debezium:build` — green, 42 unit tests, including checkstyle and spotbugs * `:sdks:java:io:debezium:integrationTest` — green, 4/4 (testcontainers MySQL + Postgres) * `:sdks:java:io:debezium:expansion-service:build` — green, shadow jar contents verified * First commit verified to compile standalone against master's dependencies Python and Go cross-language ITs have had their expected version strings updated but were not run locally, as they need the full expansion infrastructure; relying on CI for those. ### CHANGES.md Intentionally not touched yet. Proposed entry once this is approved: > * Upgraded Debezium to 3.5.2.Final in DebeziumIO, aligning with Kafka Connect API 4.1.2 (Java) ([#39568](https://github.com/apache/beam/issues/39568)). ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
