DanielLeens opened a new pull request, #13070:
URL: https://github.com/apache/seatunnel/pull/13070
### Purpose of this pull request
Add MySQL 9.0 support to the MySQL-CDC source, including the new `VECTOR`
column type, on the Java 11 baseline.
Debezium 3.x cannot be adopted directly: its parser jar and connector
modules target Java 17, and SeaTunnel builds on Java 11 with Debezium 1.9.8.
This PR uses the latest Debezium 3.x as the reference and backports what it
changed for MySQL 9.0. Upstream, those changes were test configuration
(DBZ-8030) plus `VECTOR` support (DBZ-8157).
- **Binlog client:** replace `com.zendesk:mysql-binlog-connector-java`
0.27.2 with `io.debezium:mysql-binlog-connector-java` 0.40.2 in
`connector-cdc-mysql`. It keeps the `com.github.shyiko` packages and Java 8
bytecode, and it adds column type 242, which 0.27.2 cannot deserialize.
- **DDL parsing:** the 1.9.8 grammar has no `VECTOR` token, so
`MySqlVectorTypeLexer` retypes `VECTOR` to the `VARBINARY` token, whose rule
takes the same optional `(n)`. The token text stays `VECTOR`, so the column
keeps its real type name. Identifiers merely named `vector` are not retyped.
- **Type mapping:** `VECTOR` maps to `BYTES` holding the stored
little-endian floats, 4 bytes per dimension, in `MySqlTypeUtils` and
`MySqlTypeConverter`.
- **Snapshot read:** `VECTOR` is read with `rs.getBytes()`, because the
Connector/J 8.0.x source has no decode path for this wire type in
`rs.getObject()`.
### Does this PR introduce _any_ user-facing change?
Yes.
- MySQL-CDC gains MySQL 9.0 support, and tables with `VECTOR` columns can be
captured. Docs now list 8.4.x and 9.0.x.
- `VECTOR` is delivered as `BYTES`, not `VECTOR_FLOAT_TYPE`. Mapping to the
float vector type needs a shared `cdc-base` row converter change and is out of
scope.
- `MySqlTypeConverter` is shared with the JDBC MySQL paths, which now map
`VECTOR` to `BYTES` instead of rejecting it.
### Compatibility and risk
- The binlog client change applies to every MySQL CDC job, on all server
versions. A public API comparison of 0.27.2 and 0.40.2 found two changed
members (`BinaryLogClient.gtid`,
`TableMapEventMetadataDeserializer.deserialize`), neither used by Debezium
1.9.8 or SeaTunnel. The patched `BufferedSocketInputStream` has the same public
API. The existing MySQL ITs are the regression check.
- New compile dependency: `com.github.luben:zstd-jni`, from the new binlog
client.
- No checkpoint, state or config format changes.
### How was this patch tested?
- Unit tests in `CustomMySqlAntlrDdlParserTest` and `MySqlTypeConverterTest`.
- New `Mysql9_0CDCIT`, running the shared MySQL CDC suite on MySQL 9.0.1,
plus `testMysqlCdcVectorColumn`, which asserts exact stored bytes through
snapshot and binlog.
- Not verified locally: only Spotless was run. Compilation and all tests
rely on GitHub CI for this head.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
