Minreng Wu created SPARK-62118:
----------------------------------

             Summary: Allow emitting literal non-BMP Unicode characters in JSON 
expression output
                 Key: SPARK-62118
                 URL: https://issues.apache.org/jira/browse/SPARK-62118
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Minreng Wu


When a Spark JSON expression function returns a structural value (an object or 
an array), any non-BMP Unicode character inside it is re-serialized as an 
escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes 
back as the 12-character escape {{\uD83C\uDF0A}} instead of the character 
itself.

Both forms are valid JSON and decode to the same value, so consumers that parse 
the result are unaffected. The difference shows up only for consumers that 
treat the returned text as-is, such as a before/after string comparison that 
fails because the extracted string no longer matches the original.

*Reproduction*

{code:sql}
-- structural values are affected
SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}

SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}

-- scalar strings on simple paths, and to_json, already emit the characters 
literally
SELECT get_json_object('{"a":"🌊"}', '$.a');
-- Current output: 🌊 (unchanged when the configuration is enabled)

SELECT to_json(named_struct('b', '🌊'));
-- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
{code}

*Proposal*

We propose adding a SQL configuration, 
{{spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled}}, which defaults to 
false and therefore preserves the current output. When enabled, Spark turns on 
Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}} for the 
factory that serializes JSON expression results, so the affected paths above 
emit the characters themselves. This brings them in line with the paths that 
already return these characters literally, such as simple-path 
{{get_json_object}} results, {{json_tuple}} string fields, and {{to_json}}.

*Supporting details:*
* Scope. The change covers the output paths that re-serialize values: 
structural results of {{get_json_object}} (including the shared multi-get 
optimization), structural fields of {{json_tuple}}, and values serialized by 
{{JSON_TABLE}} and {{json_query}}. JSON reads ({{from_json}}, the JSON data 
source) are out of scope and unchanged.
* Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate pairs 
by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the combined 
4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18 (master 
uses 2.22.4); whether it is suitable for the older Jackson versions on 
maintenance branches is left for the backport PRs to verify.
* Session semantics. The setting applies per session and behaves consistently 
across all evaluation paths of the affected functions, including the shared 
multi-get optimization.
* Parsing is unaffected: the input JSON is decoded the same way either way, and 
Spark already accepts both escape forms on input.
* Since the default is false, no migration guide is required.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to