[
https://issues.apache.org/jira/browse/FLINK-41204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib updated FLINK-41204:
---------------------------------
Priority: Minor (was: Major)
> VARIANT JSON parsing replaces unpaired surrogates with '?' instead of failing
> -----------------------------------------------------------------------------
>
> Key: FLINK-41204
> URL: https://issues.apache.org/jira/browse/FLINK-41204
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Minor
>
> A JSON string can hold an escaped surrogate without its pair, such as U+D800
> on its own. That is not valid Unicode, so it has no UTF-8 encoding, see RFC
> 8259 section 7. The VARIANT builder encodes text with
> {{String.getBytes(UTF_8)}}, which writes {{?}} in place of the surrogate. The
> VARIANT then holds different text than the input, and nothing fails.
> Two inputs show it:
> {noformat}
> 1. {"a\ud800": 1} lone surrogate in a key
> 2. {"a": "x\ud800"} lone surrogate in a string value
> {noformat}
> Whether Flink rejects them depends on the entry point:
> ||Entry point||Parser input||Input 1||Input 2||
> |{{PARSE_JSON}}, {{TRY_PARSE_JSON}}|bytes|fails|value stored as {{x?}}|
> |{{json}} format, VARIANT column|bytes for the record, then a {{String}} per
> VARIANT|fails|value stored as {{x?}}|
> |{{raw}} format, VARIANT|{{String}}|key stored as {{a?}}|value stored as
> {{x?}}|
> Jackson's byte parser checks surrogate pairs in field names, but not in
> string values. Its {{String}} parser checks neither. The table comes from the
> parser calls each entry point makes, with the shaded Jackson 2.22.2 on master.
> *Proposal*
> Reject unpaired surrogates in {{BinaryVariantInternalBuilder.buildJson}}, in
> keys and in string values, before they are encoded. All entry points then
> agree: {{PARSE_JSON}} fails, {{TRY_PARSE_JSON}} returns NULL, and the
> {{json}} and {{raw}} formats report a deserialization error.
> Spark made the same change in SPARK-56654 for Spark 4.3.0. It rejects lone
> surrogates in keys and string values when parsing JSON into a variant:
> {{parse_json}} fails and {{try_parse_json}} returns NULL. The check is on by
> default, and the internal conf
> {{spark.sql.variant.validateUnicodeInJsonParsing}} turns it off.
> *Open question*
> Rejecting turns input that parses today into an error. A source that reads
> such records with the {{raw}} format then fails, and so does the {{json}}
> format unless {{json.ignore-parse-errors}} skips them. Should Flink keep an
> opt-out like Spark's?
> *Out of scope*
> {{BinaryVariantBuilder}} and {{Variant.getField}} keep encoding a lone
> surrogate as {{?}}. {{StringData}} does the same, see
> {{StringUtf8Utils.encodeUTF8}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
