raminqaf commented on code in PR #29010:
URL: https://github.com/apache/flink/pull/29010#discussion_r3853073461
##########
flink-table/flink-table-runtime/src/main/java/org/apache/flink/formats/raw/RawFormatDeserializationSchema.java:
##########
@@ -278,6 +283,42 @@ public Object convert(byte[] data) {
};
}
+ /**
+ * Creates a converter that reads the bytes as a JSON document and parses
it into a {@link
+ * Variant}. Duplicate object keys are rejected, matching the default of
{@code PARSE_JSON}.
+ */
+ private static DeserializationRuntimeConverter createVariantConverter(
+ final String charsetName) {
+ // this also checks the charsetName is valid
+ final Charset configured = Charset.forName(charsetName);
+ final boolean parseFromBytes =
configured.equals(StandardCharsets.UTF_8);
+
+ return new DeserializationRuntimeConverter() {
+ private static final long serialVersionUID = 1L;
+ private transient Charset charset;
+
+ @Override
+ public void open() {
+ charset = Charset.forName(charsetName);
+ }
+
+ @Override
+ public Object convert(byte[] data) {
+ try {
+ return parseFromBytes
+ ? BinaryVariantInternalBuilder.parseJson(data,
false)
+ : BinaryVariantInternalBuilder.parseJson(
+ new String(data, charset), false);
Review Comment:
Good catch, and no, that wasn't intentional. You read it right.
The byte path handed raw bytes to Jackson, whose `createParser(byte[])` runs
its own RFC 4627 encoding auto-detection. I confirmed it parses UTF-16 bytes
even when raw.charset is UTF-8, so the declared charset was not enforced on
that path, while every non-UTF-8 charset was.
Fixed by always decoding with the declared charset before parsing. UTF-8 is
now enforced exactly like every other charset, and read is symmetric with
write.
I also dropped the now-unused `parseJson(byte[])` overload, so this no
longer touches flink-core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
