Ramin Gharib created FLINK-41305:
------------------------------------
Summary: Require UTF-8 for a VARIANT column in the raw format and
parse it from bytes
Key: FLINK-41305
URL: https://issues.apache.org/jira/browse/FLINK-41305
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
SQL / Runtime
Reporter: Ramin Gharib
Assignee: Ramin Gharib
The {{raw}} format reads a VARIANT column by decoding the message with
{{raw.charset}} into a {{String}} and parsing that text. With
{{raw.line-delimiter}} it decodes the whole message, splits it, encodes each
piece back to bytes and decodes it again before parsing. {{PARSE_JSON}} parses
UTF-8 bytes directly.
The two paths disagree on invalid UTF-8:
{noformat}
input bytes raw VARIANT PARSE_JSON
{"a":"<0xFF>"} {"a":"�"} JsonParseException: Invalid UTF-8 start byte
0xff
{noformat}
The {{raw}} format replaces the invalid byte with U+FFFD and stores it.
*Proposal*
# A VARIANT column in the {{raw}} format requires {{raw.charset}} = UTF-8. Any
other charset fails with a {{ValidationException}} when a query reads or writes
the table, not on the first record. RFC 8259 §8.1 requires UTF-8 for JSON
exchanged between systems, and {{PARSE_JSON}} already reads only UTF-8.
# The VARIANT converter parses the message bytes with
{{BinaryVariantInternalBuilder.parseJson(byte[], boolean)}}. With
{{raw.line-delimiter}}, the message is split on the UTF-8 bytes of the
delimiter, so it is never decoded into a {{String}}. UTF-8 is
self-synchronizing, so the delimiter cannot match inside a character.
# Invalid UTF-8 fails the record, as it does in {{PARSE_JSON}}.
JSON in another charset can still be read: declare the column as {{STRING}}
with {{raw.charset}}, then call {{PARSE_JSON}} on it.
*Compatibility*
VARIANT support in the {{raw}} format is new in 2.4.0 (FLINK-40241) and not
released yet. If this lands in 2.4.0 too, the restriction breaks no running job.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
