[
https://issues.apache.org/jira/browse/FLINK-41305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41305:
-----------------------------------
Labels: pull-request-available (was: )
> Require UTF-8 for a VARIANT column in the raw format and parse it from bytes
> ----------------------------------------------------------------------------
>
> Key: FLINK-41305
> URL: https://issues.apache.org/jira/browse/FLINK-41305
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Runtime
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Labels: pull-request-available
>
> The {{raw}} format reads a VARIANT column by decoding the message with
> {{raw.charset}} into a {{String}} and parsing that text. With
> {{raw.line-delimiter}} it decodes the whole message, splits it, encodes each
> piece back to bytes and decodes it again before parsing. {{PARSE_JSON}}
> parses UTF-8 bytes directly.
> The two paths disagree on invalid UTF-8:
> {noformat}
> input bytes raw VARIANT PARSE_JSON
> {"a":"<0xFF>"} {"a":"\uFFFD"} JsonParseException: Invalid UTF-8 start
> byte 0xff
> {noformat}
> The {{raw}} format replaces the invalid byte with U+FFFD and stores it.
> *Proposal*
> # A VARIANT column in the {{raw}} format requires {{raw.charset}} = UTF-8.
> Any other charset fails with a {{ValidationException}} when a query reads or
> writes the table, not on the first record. RFC 8259 ยง8.1 requires UTF-8 for
> JSON exchanged between systems, and {{PARSE_JSON}} already reads only UTF-8.
> # The VARIANT converter parses the message bytes with
> {{BinaryVariantInternalBuilder.parseJson(byte[], boolean)}}. With
> {{raw.line-delimiter}}, the message is split on the UTF-8 bytes of the
> delimiter, so it is never decoded into a {{String}}. UTF-8 is
> self-synchronizing, so the delimiter cannot match inside a character.
> # Invalid UTF-8 fails the record, as it does in {{PARSE_JSON}}.
> JSON in another charset can still be read: declare the column as {{STRING}}
> with {{raw.charset}}, then call {{PARSE_JSON}} on it.
> *Compatibility*
> VARIANT support in the {{raw}} format is new in 2.4.0 (FLINK-40241) and not
> released yet. If this lands in 2.4.0 too, the restriction breaks no running
> job.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
