[
https://issues.apache.org/jira/browse/SPARK-62118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Minreng Wu updated SPARK-62118:
-------------------------------
Description:
When a Spark JSON expression function returns a structural value (an object or
an array), any non-BMP Unicode character inside it is re-serialized as an
escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes
back as the 12-character escape {{\uD83C\uDF0A}} instead of the character
itself.
Both forms are valid JSON and decode to the same value, so consumers that parse
the result are unaffected. The difference shows up only for consumers that
treat the returned text as-is. For example, a string comparison or a hash check
against the original fails because the extracted text is no longer identical.
*Reproduction*
{code:sql}
-- structural values are affected
SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}
SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}
-- scalar strings on simple paths, and to_json, already emit the characters
literally
SELECT get_json_object('{"a":"🌊"}', '$.a');
-- Current output: 🌊 (unchanged when the configuration is enabled)
SELECT to_json(named_struct('b', '🌊'));
-- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
{code}
*Proposal*
We propose adding a SQL configuration,
{{{}spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled{}}}, which defaults
to false and therefore preserves the current output. When enabled, Spark turns
on Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}} for the
factory that serializes JSON expression results, so the affected paths above
emit the characters themselves. This brings them in line with the paths that
already return these characters literally, such as simple-path
{{get_json_object}} results, {{json_tuple}} string fields, and {{{}to_json{}}}.
*Supporting details:*
* Scope. The change covers the output paths that re-serialize values:
structural results of {{get_json_object}} (including the shared multi-get
optimization), structural fields of {{{}json_tuple{}}}, and values serialized
by {{JSON_TABLE}} and {{{}json_query{}}}. JSON reads ({{{}from_json{}}}, the
JSON data source) are out of scope and unchanged.
* Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate pairs
by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the combined
4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18 (master
uses 2.22.4); whether it is suitable for the older Jackson versions on
maintenance branches is left for the backport PRs to verify.
* Session semantics. The setting applies per session and behaves consistently
across all evaluation paths of the affected functions, including the shared
multi-get optimization.
* Parsing is unaffected: the input JSON is decoded the same way either way,
and Spark already accepts both escape forms on input.
* Since the default is false, no migration guide is required.
was:
When a Spark JSON expression function returns a structural value (an object or
an array), any non-BMP Unicode character inside it is re-serialized as an
escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes
back as the 12-character escape {{\uD83C\uDF0A}} instead of the character
itself.
Both forms are valid JSON and decode to the same value, so consumers that parse
the result are unaffected. The difference shows up only for consumers that
treat the returned text as-is, such as a before/after string comparison that
fails because the extracted string no longer matches the original.
*Reproduction*
{code:sql}
-- structural values are affected
SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}
SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
-- Current output: {"s":"\uD83C\uDF0A"}
-- Output with the configuration enabled: {"s":"🌊"}
-- scalar strings on simple paths, and to_json, already emit the characters
literally
SELECT get_json_object('{"a":"🌊"}', '$.a');
-- Current output: 🌊 (unchanged when the configuration is enabled)
SELECT to_json(named_struct('b', '🌊'));
-- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
{code}
*Proposal*
We propose adding a SQL configuration,
{{spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled}}, which defaults to
false and therefore preserves the current output. When enabled, Spark turns on
Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}} for the
factory that serializes JSON expression results, so the affected paths above
emit the characters themselves. This brings them in line with the paths that
already return these characters literally, such as simple-path
{{get_json_object}} results, {{json_tuple}} string fields, and {{to_json}}.
*Supporting details:*
* Scope. The change covers the output paths that re-serialize values:
structural results of {{get_json_object}} (including the shared multi-get
optimization), structural fields of {{json_tuple}}, and values serialized by
{{JSON_TABLE}} and {{json_query}}. JSON reads ({{from_json}}, the JSON data
source) are out of scope and unchanged.
* Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate pairs
by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the combined
4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18 (master
uses 2.22.4); whether it is suitable for the older Jackson versions on
maintenance branches is left for the backport PRs to verify.
* Session semantics. The setting applies per session and behaves consistently
across all evaluation paths of the affected functions, including the shared
multi-get optimization.
* Parsing is unaffected: the input JSON is decoded the same way either way, and
Spark already accepts both escape forms on input.
* Since the default is false, no migration guide is required.
> Allow emitting literal non-BMP Unicode characters in JSON expression output
> ---------------------------------------------------------------------------
>
> Key: SPARK-62118
> URL: https://issues.apache.org/jira/browse/SPARK-62118
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Minreng Wu
> Priority: Minor
>
> When a Spark JSON expression function returns a structural value (an object
> or an array), any non-BMP Unicode character inside it is re-serialized as an
> escaped UTF-16 surrogate pair. For example, a JSON object containing 🌊 comes
> back as the 12-character escape {{\uD83C\uDF0A}} instead of the character
> itself.
> Both forms are valid JSON and decode to the same value, so consumers that
> parse the result are unaffected. The difference shows up only for consumers
> that treat the returned text as-is. For example, a string comparison or a
> hash check against the original fails because the extracted text is no longer
> identical.
> *Reproduction*
> {code:sql}
> -- structural values are affected
> SELECT get_json_object('{"a":{"s":"🌊"}}', '$.a');
> -- Current output: {"s":"\uD83C\uDF0A"}
> -- Output with the configuration enabled: {"s":"🌊"}
> SELECT get_json_object('{"a":[{"s":"🌊"}]}', '$.a[0]');
> -- Current output: {"s":"\uD83C\uDF0A"}
> -- Output with the configuration enabled: {"s":"🌊"}
> -- scalar strings on simple paths, and to_json, already emit the characters
> literally
> SELECT get_json_object('{"a":"🌊"}', '$.a');
> -- Current output: 🌊 (unchanged when the configuration is enabled)
> SELECT to_json(named_struct('b', '🌊'));
> -- Current output: {"b":"🌊"} (unchanged when the configuration is enabled)
> {code}
> *Proposal*
> We propose adding a SQL configuration,
> {{{}spark.sql.json.combineUnicodeSurrogatesInUtf8.enabled{}}}, which defaults
> to false and therefore preserves the current output. When enabled, Spark
> turns on Jackson's {{JsonWriteFeature.COMBINE_UNICODE_SURROGATES_IN_UTF8}}
> for the factory that serializes JSON expression results, so the affected
> paths above emit the characters themselves. This brings them in line with the
> paths that already return these characters literally, such as simple-path
> {{get_json_object}} results, {{json_tuple}} string fields, and
> {{{}to_json{}}}.
> *Supporting details:*
> * Scope. The change covers the output paths that re-serialize values:
> structural results of {{get_json_object}} (including the shared multi-get
> optimization), structural fields of {{{}json_tuple{}}}, and values serialized
> by {{JSON_TABLE}} and {{{}json_query{}}}. JSON reads ({{{}from_json{}}}, the
> JSON data source) are out of scope and unchanged.
> * Jackson behavior. Jackson's byte-based UTF-8 writers escape surrogate
> pairs by default; {{COMBINE_UNICODE_SURROGATES_IN_UTF8}} makes them emit the
> combined 4-byte UTF-8 sequence instead. The feature exists since Jackson 2.18
> (master uses 2.22.4); whether it is suitable for the older Jackson versions
> on maintenance branches is left for the backport PRs to verify.
> * Session semantics. The setting applies per session and behaves
> consistently across all evaluation paths of the affected functions, including
> the shared multi-get optimization.
> * Parsing is unaffected: the input JSON is decoded the same way either way,
> and Spark already accepts both escape forms on input.
> * Since the default is false, no migration guide is required.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]