Pika-Yalei opened a new pull request, #4603: URL: https://github.com/apache/flink-cdc/pull/4603
## What is the purpose of this pull request? Fix [FLINK-36263](https://issues.apache.org/jira/browse/FLINK-36263): Canal JSON UPDATE messages currently include every column in `old`, rather than only the columns whose values changed. ## Brief change log - Build the Canal JSON envelope with Flink's native row-to-JSON converters and remove unchanged fields from `old` using structural JSON equality. - Preserve the complete `data` row, configured null-field handling, and existing timestamp, map-key and decimal serialization options. - Keep INSERT and DELETE behavior unchanged. - Add regression coverage for null-to-value/value-to-null changes, entirely unchanged rows, multiple changed complex values, independently constructed equal values and map ordering, consecutive events, schema additions, and REPLACE events with a before image. ## Verifying this change The updated Canal serializer tests failed against the original implementation (5 tests, 5 assertion failures, 0 errors), then passed with the fix. All seven Kafka connector unit-test classes passed in each configuration (24 tests, 0 failures, 0 errors, 0 skipped): - JDK 11 / Flink 1.20.3: `test` - JDK 17 / Flink 1.20.3: `clean test` - JDK 17 / Flink 2.2.0: `-Pflink2 clean package` Commands use: ```sh mvn -pl :flink-cdc-pipeline-connector-kafka -am \ -Dtest=JsonSerializationSchemaTest,CsvSerializationSchemaTest,KafkaDataSinkFactoryTest,RecordDataConverterTest,TableSchemaInfoTest,CanalJsonSerializationSchemaTest,DebeziumJsonSerializationSchemaTest \ -Dsurefire.failIfNoSpecifiedTests=false test ``` For the JDK 17 runs, replace the final goal with `clean test` or `-Pflink2 clean package`, respectively. The Flink 2 profile requires packaging the compatibility artifact before the downstream dependency-copy execution. Spotless formatting/checks, Checkstyle, and `git diff --check` passed. Docker-based integration/end-to-end tests have not been run (Docker is unavailable in the development environment). Upstream CI remains to be verified; this PR is intentionally a draft. ## Documentation - Does this pull request introduce a new feature? No; it corrects the existing Canal JSON UPDATE representation. - If yes, how is the feature documented? Not applicable. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — OpenAI Codex. The exact model version is not exposed by the execution environment. Generated-by: OpenAI Codex (model version not exposed) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
