[
https://issues.apache.org/jira/browse/FLINK-41305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138059#comment-18138059
]
Yuepeng Pan edited comment on FLINK-41305 at 11/4/26, 3:46 PM:
---------------------------------------------------------------
Merged into master(2.5.0) via:
2180171f1e2544f36f92cf694b05b6233f13ed15
33a8167b49a17c6f45e0e3f864c9de859aa36430
Merged into release-2.4 via:
110e2d4269488958fe79ba9f840e9774c0474a9d
c6b2757cf12fcfbe99d42e2866fefb91cfa6b827
was (Author: JIRAUSER301104):
Merged into master(2.5.0) via:
2180171f1e2544f36f92cf694b05b6233f13ed15
33a8167b49a17c6f45e0e3f864c9de859aa36430
> Require UTF-8 for a VARIANT column in the raw format and parse it from bytes
> ----------------------------------------------------------------------------
>
> Key: FLINK-41305
> URL: https://issues.apache.org/jira/browse/FLINK-41305
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Runtime
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Labels: pull-request-available
>
> The {{raw}} format reads a VARIANT column by decoding the message with
> {{raw.charset}} into a {{String}} and parsing that text. With
> {{raw.line-delimiter}} it decodes the whole message, splits it, encodes each
> piece back to bytes and converts it, for any column type. {{PARSE_JSON}}
> parses UTF-8 bytes directly.
> The two paths disagree on invalid UTF-8:
> {noformat}
> input bytes raw VARIANT PARSE_JSON
> {"a":"<0xFF>"} {"a":"\uFFFD"} JsonParseException: Invalid UTF-8 start
> byte 0xff
> {noformat}
> The {{raw}} format replaces the invalid byte with U+FFFD and stores it. With
> a delimiter, a STRING column loses invalid bytes the same way, although it
> keeps them without a delimiter.
> *Proposal*
> # A VARIANT column in the {{raw}} format requires {{raw.charset}} = UTF-8.
> Any other charset fails with a {{ValidationException}} when a query reads or
> writes the table, not on the first record. RFC 8259 ยง8.1 requires UTF-8 for
> JSON exchanged between systems, and {{PARSE_JSON}} already reads only UTF-8.
> # The VARIANT converter parses the message bytes with
> {{BinaryVariantInternalBuilder.parseJson(byte[], boolean)}}. Invalid UTF-8
> fails the record, as it does in {{PARSE_JSON}}.
> # {{raw.line-delimiter}} is supported only for a CHAR, VARCHAR or STRING
> column. Any other column type fails with a {{ValidationException}} when a
> query reads or writes the table, because the bytes of a number, BYTES or
> VARIANT value can contain the delimiter. Newline-delimited JSON can be read
> through a STRING column and {{PARSE_JSON}}.
> # If {{raw.charset}} is UTF-8, the message is split on the UTF-8 bytes of the
> delimiter and never decoded. UTF-8 is self-synchronizing, so the delimiter
> cannot match inside a character. Other charsets keep the decoded split,
> because their delimiter bytes can match inside a character, as {{|}} does in
> GBK.
> JSON in another charset can still be read: declare the column as {{STRING}}
> with {{raw.charset}}, then call {{PARSE_JSON}} on it.
> *Compatibility*
> VARIANT support in the {{raw}} format (FLINK-40241) and
> {{raw.line-delimiter}} (FLINK-39401) are new in 2.4.0 and not released yet.
> If this lands in 2.4.0 too, the restrictions break no running job.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
