Ramin Gharib created FLINK-41204:
------------------------------------
Summary: VARIANT JSON parsing replaces unpaired surrogates with
'?' instead of failing
Key: FLINK-41204
URL: https://issues.apache.org/jira/browse/FLINK-41204
Project: Flink
Issue Type: Bug
Components: API / Core
Reporter: Ramin Gharib
A JSON string can hold an escaped surrogate without its pair, such as U+D800 on
its own. That is not valid Unicode, so it has no UTF-8 encoding, see RFC 8259
section 7. The VARIANT builder encodes text with {{String.getBytes(UTF_8)}},
which writes {{?}} in place of the surrogate. The VARIANT then holds different
text than the input, and nothing fails.
Two inputs show it:
{noformat}
1. {"a\ud800": 1} lone surrogate in a key
2. {"a": "x\ud800"} lone surrogate in a string value
{noformat}
Whether Flink rejects them depends on the entry point:
||Entry point||Parser input||Input 1||Input 2||
|{{PARSE_JSON}}, {{TRY_PARSE_JSON}}|bytes|fails|value stored as {{x?}}|
|{{json}} format, VARIANT column|bytes for the record, then a {{String}} per
VARIANT|fails|value stored as {{x?}}|
|{{raw}} format, VARIANT|{{String}}|key stored as {{a?}}|value stored as {{x?}}|
Jackson's byte parser checks surrogate pairs in field names, but not in string
values. Its {{String}} parser checks neither. The table comes from the parser
calls each entry point makes, with the shaded Jackson 2.22.2 on master.
*Proposal*
Reject unpaired surrogates in {{BinaryVariantInternalBuilder.buildJson}}, in
keys and in string values, before they are encoded. All entry points then
agree: {{PARSE_JSON}} fails, {{TRY_PARSE_JSON}} returns NULL, and the {{json}}
and {{raw}} formats report a deserialization error.
Spark made the same change in SPARK-56654 for Spark 4.3.0. It rejects lone
surrogates in keys and string values when parsing JSON into a variant:
{{parse_json}} fails and {{try_parse_json}} returns NULL. The check is on by
default, and the internal conf
{{spark.sql.variant.validateUnicodeInJsonParsing}} turns it off.
*Open question*
Rejecting turns input that parses today into an error. A source that reads such
records with the {{raw}} format then fails, and so does the {{json}} format
unless {{json.ignore-parse-errors}} skips them. Should Flink keep an opt-out
like Spark's?
*Out of scope*
{{BinaryVariantBuilder}} and {{Variant.getField}} keep encoding a lone
surrogate as {{?}}. {{StringData}} does the same, see
{{StringUtf8Utils.encodeUTF8}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
