[ 
https://issues.apache.org/jira/browse/SPARK-62118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Minreng Wu updated SPARK-62118:
-------------------------------
    Description: 
When a Spark JSON expression function returns a structural value (an object or 
an array), any non-BMP Unicode character inside it is re-serialized as an 
escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes 
back as the 12-character escape {{\uD83C\uDF0A}} instead of the character 
itself.

Both forms are valid JSON and decode to the same value, so consumers that parse 
the result are unaffected. The difference shows up only for consumers that 
treat the returned text as-is. For example, a string comparison or a hash check 
against the original fails because the extracted text is no longer identical.

*Reproduction*
{code:sql}
-- structural values are affected
SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}

SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}

-- scalar strings on simple paths, and to_json, already emit the characters 
literally
SELECT get_json_object('{"a":"🌊"}', '$.a');
-- Current output: 🌊 (unchanged when the configuration is enabled)

SELECT to_json(named_struct('b', '🌊'));
-- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
{code}
*Proposal*

We propose adding a SQL configuration, 
{{{}spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled{}}}, which defaults 
to false and therefore preserves the current output. When enabled, Spark turns 
on Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}} for the 
factory that serializes JSON expression results, so the affected paths above 
emit the characters themselves. This brings them in line with the paths that 
already return these characters literally, such as simple-path 
{{get_json_object}} results, {{json_tuple}} string fields, and {{{}to_json{}}}.

*Supporting details:*
 * Scope. The change covers the output paths that re-serialize values: 
structural results of {{get_json_object}} (including the shared multi-get 
optimization), structural fields of {{{}json_tuple{}}}, and values serialized 
by {{JSON_TABLE}} and {{{}json_query{}}}. JSON reads ({{{}from_json{}}}, the 
JSON data source) are out of scope and unchanged.
 * Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate pairs 
by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the combined 
4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18 (master 
uses 2.22.4); whether it is suitable for the older Jackson versions on 
maintenance branches is left for the backport PRs to verify.
 * Session semantics. The setting applies per session and behaves consistently 
across all evaluation paths of the affected functions, including the shared 
multi-get optimization.
 * Parsing is unaffected: the input JSON is decoded the same way either way, 
and Spark already accepts both escape forms on input.
 * Since the default is false, no migration guide is required.

  was:
When a Spark JSON expression function returns a structural value (an object or 
an array), any non-BMP Unicode character inside it is re-serialized as an 
escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes 
back as the 12-character escape {{\uD83C\uDF0A}} instead of the character 
itself.

Both forms are valid JSON and decode to the same value, so consumers that parse 
the result are unaffected. The difference shows up only for consumers that 
treat the returned text as-is, such as a before/after string comparison that 
fails because the extracted string no longer matches the original.

*Reproduction*

{code:sql}
-- structural values are affected
SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}

SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}

-- scalar strings on simple paths, and to_json, already emit the characters 
literally
SELECT get_json_object('{"a":"🌊"}', '$.a');
-- Current output: 🌊 (unchanged when the configuration is enabled)

SELECT to_json(named_struct('b', '🌊'));
-- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
{code}

*Proposal*

We propose adding a SQL configuration, 
{{spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled}}, which defaults to 
false and therefore preserves the current output. When enabled, Spark turns on 
Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}} for the 
factory that serializes JSON expression results, so the affected paths above 
emit the characters themselves. This brings them in line with the paths that 
already return these characters literally, such as simple-path 
{{get_json_object}} results, {{json_tuple}} string fields, and {{to_json}}.

*Supporting details:*
* Scope. The change covers the output paths that re-serialize values: 
structural results of {{get_json_object}} (including the shared multi-get 
optimization), structural fields of {{json_tuple}}, and values serialized by 
{{JSON_TABLE}} and {{json_query}}. JSON reads ({{from_json}}, the JSON data 
source) are out of scope and unchanged.
* Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate pairs 
by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the combined 
4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18 (master 
uses 2.22.4); whether it is suitable for the older Jackson versions on 
maintenance branches is left for the backport PRs to verify.
* Session semantics. The setting applies per session and behaves consistently 
across all evaluation paths of the affected functions, including the shared 
multi-get optimization.
* Parsing is unaffected: the input JSON is decoded the same way either way, and 
Spark already accepts both escape forms on input.
* Since the default is false, no migration guide is required.


> Allow emitting literal non-BMP Unicode characters in JSON expression output
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-62118
>                 URL: https://issues.apache.org/jira/browse/SPARK-62118
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Minreng Wu
>            Priority: Minor
>
> When a Spark JSON expression function returns a structural value (an object 
> or an array), any non-BMP Unicode character inside it is re-serialized as an 
> escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes 
> back as the 12-character escape {{\uD83C\uDF0A}} instead of the character 
> itself.
> Both forms are valid JSON and decode to the same value, so consumers that 
> parse the result are unaffected. The difference shows up only for consumers 
> that treat the returned text as-is. For example, a string comparison or a 
> hash check against the original fails because the extracted text is no longer 
> identical.
> *Reproduction*
> {code:sql}
> -- structural values are affected
> SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
> -- Current output: {"s":"\uD83C\uDF0A"}
> -- Output with the configuration enabled: {"s":"🌊"}
> SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
> -- Current output: {"s":"\uD83C\uDF0A"}
> -- Output with the configuration enabled: {"s":"🌊"}
> -- scalar strings on simple paths, and to_json, already emit the characters 
> literally
> SELECT get_json_object('{"a":"🌊"}', '$.a');
> -- Current output: 🌊 (unchanged when the configuration is enabled)
> SELECT to_json(named_struct('b', '🌊'));
> -- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
> {code}
> *Proposal*
> We propose adding a SQL configuration, 
> {{{}spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled{}}}, which defaults 
> to false and therefore preserves the current output. When enabled, Spark 
> turns on Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}} 
> for the factory that serializes JSON expression results, so the affected 
> paths above emit the characters themselves. This brings them in line with the 
> paths that already return these characters literally, such as simple-path 
> {{get_json_object}} results, {{json_tuple}} string fields, and 
> {{{}to_json{}}}.
> *Supporting details:*
>  * Scope. The change covers the output paths that re-serialize values: 
> structural results of {{get_json_object}} (including the shared multi-get 
> optimization), structural fields of {{{}json_tuple{}}}, and values serialized 
> by {{JSON_TABLE}} and {{{}json_query{}}}. JSON reads ({{{}from_json{}}}, the 
> JSON data source) are out of scope and unchanged.
>  * Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate 
> pairs by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the 
> combined 4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18 
> (master uses 2.22.4); whether it is suitable for the older Jackson versions 
> on maintenance branches is left for the backport PRs to verify.
>  * Session semantics. The setting applies per session and behaves 
> consistently across all evaluation paths of the affected functions, including 
> the shared multi-get optimization.
>  * Parsing is unaffected: the input JSON is decoded the same way either way, 
> and Spark already accepts both escape forms on input.
>  * Since the default is false, no migration guide is required.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to