Minreng Wu created SPARK-62118:
----------------------------------
Summary: Allow emitting literal non-BMP Unicode characters in JSON
expression output
Key: SPARK-62118
URL: https://issues.apache.org/jira/browse/SPARK-62118
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Minreng Wu
When a Spark JSON expression function returns a structural value (an object or
an array), any non-BMP Unicode character inside it is re-serialized as an
escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes
back as the 12-character escape {{\uD83C\uDF0A}} instead of the character
itself.
Both forms are valid JSON and decode to the same value, so consumers that parse
the result are unaffected. The difference shows up only for consumers that
treat the returned text as-is, such as a before/after string comparison that
fails because the extracted string no longer matches the original.
*Reproduction*
{code:sql}
-- structural values are affected
SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}
SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}
-- scalar strings on simple paths, and to_json, already emit the characters
literally
SELECT get_json_object('{"a":"🌊"}', '$.a');
-- Current output: 🌊 (unchanged when the configuration is enabled)
SELECT to_json(named_struct('b', '🌊'));
-- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
{code}
*Proposal*
We propose adding a SQL configuration,
{{spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled}}, which defaults to
false and therefore preserves the current output. When enabled, Spark turns on
Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}} for the
factory that serializes JSON expression results, so the affected paths above
emit the characters themselves. This brings them in line with the paths that
already return these characters literally, such as simple-path
{{get_json_object}} results, {{json_tuple}} string fields, and {{to_json}}.
*Supporting details:*
* Scope. The change covers the output paths that re-serialize values:
structural results of {{get_json_object}} (including the shared multi-get
optimization), structural fields of {{json_tuple}}, and values serialized by
{{JSON_TABLE}} and {{json_query}}. JSON reads ({{from_json}}, the JSON data
source) are out of scope and unchanged.
* Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate pairs
by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the combined
4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18 (master
uses 2.22.4); whether it is suitable for the older Jackson versions on
maintenance branches is left for the backport PRs to verify.
* Session semantics. The setting applies per session and behaves consistently
across all evaluation paths of the affected functions, including the shared
multi-get optimization.
* Parsing is unaffected: the input JSON is decoded the same way either way, and
Spark already accepts both escape forms on input.
* Since the default is false, no migration guide is required.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]