[
https://issues.apache.org/jira/browse/FLINK-41305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib updated FLINK-41305:
---------------------------------
Description:
The {{raw}} format reads a VARIANT column by decoding the message with
{{raw.charset}} into a {{String}} and parsing that text. With
{{raw.line-delimiter}} it decodes the whole message, splits it, encodes each
piece back to bytes and converts it, for any column type. {{PARSE_JSON}} parses
UTF-8 bytes directly.
The two paths disagree on invalid UTF-8:
{noformat}
input bytes raw VARIANT PARSE_JSON
{"a":"<0xFF>"} {"a":"\uFFFD"} JsonParseException: Invalid UTF-8 start
byte 0xff
{noformat}
The {{raw}} format replaces the invalid byte with U+FFFD and stores it. With a
delimiter, a STRING column loses invalid bytes the same way, although it keeps
them without a delimiter.
*Proposal*
# A VARIANT column in the {{raw}} format requires {{raw.charset}} = UTF-8. Any
other charset fails with a {{ValidationException}} when a query reads or writes
the table, not on the first record. RFC 8259 §8.1 requires UTF-8 for JSON
exchanged between systems, and {{PARSE_JSON}} already reads only UTF-8.
# The VARIANT converter parses the message bytes with
{{BinaryVariantInternalBuilder.parseJson(byte[], boolean)}}. Invalid UTF-8
fails the record, as it does in {{PARSE_JSON}}.
# {{raw.line-delimiter}} is supported only for a CHAR, VARCHAR or STRING
column. Any other column type fails with a {{ValidationException}} when a query
reads or writes the table, because the bytes of a number, BYTES or VARIANT
value can contain the delimiter. Newline-delimited JSON can be read through a
STRING column and {{PARSE_JSON}}.
# If {{raw.charset}} is UTF-8, the message is split on the UTF-8 bytes of the
delimiter and never decoded. UTF-8 is self-synchronizing, so the delimiter
cannot match inside a character. Other charsets keep the decoded split, because
their delimiter bytes can match inside a character, as {{|}} does in GBK.
JSON in another charset can still be read: declare the column as {{STRING}}
with {{raw.charset}}, then call {{PARSE_JSON}} on it.
*Compatibility*
VARIANT support in the {{raw}} format (FLINK-40241) and {{raw.line-delimiter}}
(FLINK-39401) are new in 2.4.0 and not released yet. If this lands in 2.4.0
too, the restrictions break no running job.
was:
The {{raw}} format reads a VARIANT column by decoding the message with
{{raw.charset}} into a {{String}} and parsing that text. With
{{raw.line-delimiter}} it decodes the whole message, splits it, encodes each
piece back to bytes and decodes it again before parsing. {{PARSE_JSON}} parses
UTF-8 bytes directly.
The two paths disagree on invalid UTF-8:
{noformat}
input bytes raw VARIANT PARSE_JSON
{"a":"<0xFF>"} {"a":"\uFFFD"} JsonParseException: Invalid UTF-8 start
byte 0xff
{noformat}
The {{raw}} format replaces the invalid byte with U+FFFD and stores it.
*Proposal*
# A VARIANT column in the {{raw}} format requires {{raw.charset}} = UTF-8. Any
other charset fails with a {{ValidationException}} when a query reads or writes
the table, not on the first record. RFC 8259 §8.1 requires UTF-8 for JSON
exchanged between systems, and {{PARSE_JSON}} already reads only UTF-8.
# The VARIANT converter parses the message bytes with
{{BinaryVariantInternalBuilder.parseJson(byte[], boolean)}}. With
{{raw.line-delimiter}}, the message is split on the UTF-8 bytes of the
delimiter, so it is never decoded into a {{String}}. UTF-8 is
self-synchronizing, so the delimiter cannot match inside a character.
# Invalid UTF-8 fails the record, as it does in {{PARSE_JSON}}.
JSON in another charset can still be read: declare the column as {{STRING}}
with {{raw.charset}}, then call {{PARSE_JSON}} on it.
*Compatibility*
VARIANT support in the {{raw}} format is new in 2.4.0 (FLINK-40241) and not
released yet. If this lands in 2.4.0 too, the restriction breaks no running job.
> Require UTF-8 for a VARIANT column in the raw format and parse it from bytes
> ----------------------------------------------------------------------------
>
> Key: FLINK-41305
> URL: https://issues.apache.org/jira/browse/FLINK-41305
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Runtime
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Labels: pull-request-available
>
> The {{raw}} format reads a VARIANT column by decoding the message with
> {{raw.charset}} into a {{String}} and parsing that text. With
> {{raw.line-delimiter}} it decodes the whole message, splits it, encodes each
> piece back to bytes and converts it, for any column type. {{PARSE_JSON}}
> parses UTF-8 bytes directly.
> The two paths disagree on invalid UTF-8:
> {noformat}
> input bytes raw VARIANT PARSE_JSON
> {"a":"<0xFF>"} {"a":"\uFFFD"} JsonParseException: Invalid UTF-8 start
> byte 0xff
> {noformat}
> The {{raw}} format replaces the invalid byte with U+FFFD and stores it. With
> a delimiter, a STRING column loses invalid bytes the same way, although it
> keeps them without a delimiter.
> *Proposal*
> # A VARIANT column in the {{raw}} format requires {{raw.charset}} = UTF-8.
> Any other charset fails with a {{ValidationException}} when a query reads or
> writes the table, not on the first record. RFC 8259 §8.1 requires UTF-8 for
> JSON exchanged between systems, and {{PARSE_JSON}} already reads only UTF-8.
> # The VARIANT converter parses the message bytes with
> {{BinaryVariantInternalBuilder.parseJson(byte[], boolean)}}. Invalid UTF-8
> fails the record, as it does in {{PARSE_JSON}}.
> # {{raw.line-delimiter}} is supported only for a CHAR, VARCHAR or STRING
> column. Any other column type fails with a {{ValidationException}} when a
> query reads or writes the table, because the bytes of a number, BYTES or
> VARIANT value can contain the delimiter. Newline-delimited JSON can be read
> through a STRING column and {{PARSE_JSON}}.
> # If {{raw.charset}} is UTF-8, the message is split on the UTF-8 bytes of the
> delimiter and never decoded. UTF-8 is self-synchronizing, so the delimiter
> cannot match inside a character. Other charsets keep the decoded split,
> because their delimiter bytes can match inside a character, as {{|}} does in
> GBK.
> JSON in another charset can still be read: declare the column as {{STRING}}
> with {{raw.charset}}, then call {{PARSE_JSON}} on it.
> *Compatibility*
> VARIANT support in the {{raw}} format (FLINK-40241) and
> {{raw.line-delimiter}} (FLINK-39401) are new in 2.4.0 and not released yet.
> If this lands in 2.4.0 too, the restrictions break no running job.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
