ODriscollBoys opened a new pull request, #10484: URL: https://github.com/apache/paimon/pull/10484
### Purpose The MySQL, PostgreSQL and MongoDB synchronization actions convert every Debezium `SourceRecord` to JSON with Kafka's `JsonConverter`, which always writes UTF-8. `CdcDebeziumDeserializationSchema` then decoded those bytes with `new String(bytes)`, i.e. the JVM default charset. When the default charset is not UTF-8 (Java 8-17 started with `LANG=C`, where it is US-ASCII, or a Windows/GBK host), every non-ASCII character in the synchronized rows is replaced or garbled before it reaches Paimon. Binary columns had the same mismatch in the other direction: `DebeziumSchemaUtils` and `PostgresRecordParser` decoded Debezium binary values into a string with the default charset, while `TypeUtils.castFromCdcValueString` encodes `VARBINARY` values back with UTF-8, so the bytes only round-tripped on UTF-8 JVMs. The Avro path also decoded `ByteBuffer.array()`, i.e. the whole backing array rather than the readable bytes of the buffer. This change: - decodes the JSON produced by `JsonConverter` as UTF-8; - decodes Debezium binary values (JSON and Avro) as UTF-8, matching the UTF-8 encoding in `castFromCdcValueString`; - decodes only the remaining bytes of an Avro `ByteBuffer`, without moving its position; - uses UTF-8 when logging invalid JSON messages in `CdcJsonDeserializationSchema` and `KafkaDebeziumJsonDeserializationSchema`. Behavior on JVMs whose default charset is already UTF-8 is unchanged. ### Tests - `CdcDebeziumDeserializationSchemaTest` (new): a record with non-ASCII string values is emitted unchanged. - `DebeziumSchemaUtilsTest#testTransformBinaryRawValueAsUtf8`: a Base64 binary value is decoded as UTF-8 and round-trips through `castFromCdcValueString`. - `DebeziumSchemaUtilsTest#testTransformAvroBytesDecodesOnlyReadableBytes`: only the readable bytes of a buffer that is a window into a larger array are decoded, and the buffer is not consumed. This one fails without the change on any JVM. With `-DextraJavaTestArgs=-Dfile.encoding=ISO-8859-1`, all three new tests fail on master with garbled output and pass with this change. The existing Debezium JSON, Avro and BSON parser tests, `KafkaDebeziumJsonDeserializationSchemaTest` and `PostgresRecordParserTest` still pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
