suntectec opened a new pull request, #4533: URL: https://github.com/apache/flink-cdc/pull/4533
## What is the purpose of this pull request? Add a DB2 pipeline connector so the YAML Pipeline API can read from IBM DB2, alongside the existing MySQL / Postgres / Oracle / SQL Server pipeline connectors. Before this change DB2 was only reachable from the DataStream / SQL APIs through `flink-connector-db2-cdc`. JIRA: [FLINK-40508](https://issues.apache.org/jira/browse/FLINK-40508) ## Brief change log - New module `flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-db2`, registered in the pipeline-connectors parent `pom.xml`. - `Db2DataSourceFactory` (identifier `db2`) plus `Db2DataSource`, `Db2PipelineSource`, `Db2EventDeserializer`, `Db2MetadataAccessor`, `Db2PipelineRecordEmitter`, `Db2SchemaDataTypeInference`, `Db2SchemaUtils` and `Db2TypeUtils`. - Startup and snapshot behaviour reuse the existing source connector: `scan.startup.mode` accepts `initial` (default) and `latest-offset`, captured tables are declared as `schema.table`. - `jdbc.properties.*` are forwarded to Debezium as `database.*` connector properties, and an explicitly configured `debezium.*` property wins. A test reads `VALUES CURRENT SCHEMA` back from a real connection to prove the properties reach JDBC. - Type mapping preserves length, precision and scale for CHAR/VARCHAR, BINARY/VARBINARY, integer, floating point, DECIMAL/NUMERIC, DATE, TIME and TIMESTAMP. Types Debezium cannot materialise reliably (`DECFLOAT`, `XML`, `CLOB`, `DBCLOB`, `BLOB`) fail fast during schema inference instead of being silently advertised, so schema and record fields cannot drift apart. - The IBM JDBC driver is excluded from the shaded artifact because of its license; installing it into `FLINK_HOME/lib` of the submitting client and every JM/TM is documented. - Capability boundaries are documented explicitly: `schema-change.enabled` only forwards schema change events reported by the underlying source and does not enable runtime DDL discovery, checkpoint/savepoint restore does not refresh an outdated schema, and UPDATE events require `CHG_UPD_TO_DEL_INS='Y'` on the capture table because direct `U` records are not supported. - `.github/workflows/modules.py` adds the module to `MODULES_DB2`, so CI builds and tests it for Flink 1.20 and Flink 2.x; `.github/labeler.yml` adds the `db2-pipeline-connector` rule. - Documentation in `docs/content` and `docs/i18n/zh`, plus the pipeline-connectors overview pages and the Docusaurus sidebar. `flink-connector-db2-cdc` is not modified. `src/test/resources/db2_server/` is copied from the source connector because `Db2TestBase` resolves the Docker build context through `Paths.get(url.toURI())`, which cannot read resources out of a test-jar. ## Verifying this change This change added tests and can be verified as follows: - `Db2TypeUtilsTest` covers the supported mappings, length and precision handling, and the rejection path for unsupported types. - `Db2DataSourceFactoryITCase` (13 cases) runs against a real DB2 instance through Testcontainers, including `jdbc.properties.*` / `debezium.*` precedence. - `Db2PipelineITCase` (5 cases) covers `initial` with and without schema changes, an 18-column full-type table for snapshot plus streaming INSERT/UPDATE/DELETE, and fail-fast on the `UNSUPPORTED_TYPES` fixture. - `mvn verify` passes on Java 11 / Flink 1.20.3 and Java 17 / Flink 2.2.0 (`-Pflink2`): 26 tests per generation, plus Spotless and Checkstyle. - Documentation verified with `npm test` and `npm run build` for both locales. ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Qoder) Generated-by: Qoder -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
