taoran92 opened a new pull request, #11046:
URL: https://github.com/apache/paimon/pull/11046
### Purpose
Fixes #11045.
Kafka Connect's `JsonConverter` emits UTF-8 JSON bytes, but the CDC
deserializer decoded them using the JVM's default charset. This corrupts
non-ASCII text on non-UTF-8 JVMs, including MySQL snapshot and binlog records.
Decode these bytes explicitly as UTF-8, and use the same charset when logging
invalid JSON in the JSON and Kafka Debezium deserializers.
### Tests
- Add a regression test that starts fresh JVMs with UTF-8 and US-ASCII
default charsets and checks non-ASCII text and escaped characters with schema
inclusion enabled and disabled.
- Passed the targeted Maven tests and standard validation checks:
```shell
mvn -o -B -ntp -pl paimon-flink/paimon-flink-cdc -Pflink1 \
-DwildcardSuites=none \
-Dtest=CdcDebeziumDeserializationSchemaTest,KafkaDebeziumJsonDeserializationSchemaTest
test
```
- Verified the regression test entry point on JDK 8 and JDK 11: the parent
commit fails with US-ASCII; the current commit passes with both charsets. All
changed Java files also compile with `javac --release 8`.
- MySQL/Flink end-to-end validation was not run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]