[
https://issues.apache.org/jira/browse/SPARK-57990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved SPARK-57990.
------------------------------
Fix Version/s: (was: 4.3.0)
Resolution: Fixed
Issue resolved by pull request 57069
[https://github.com/apache/spark/pull/57069]
> Extend shared get_json_object parsing to array-index paths and
> object-or-array fallbacks
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-57990
> URL: https://issues.apache.org/jira/browse/SPARK-57990
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 5.0.0
>
>
> SPARK-47670 introduced opt-in shared parsing for repeated {{get_json_object}}
> calls over the same JSON input. SPARK-57626 extended it from top-level fields
> to literal nested named object paths. Array subscripts are still
> intentionally excluded, and the optimizer does not traverse conditional
> expressions, so a common object-or-array fallback pattern still reparses the
> same input for each projected field.
> {code:sql}
> SELECT
> coalesce(
> get_json_object(json, '$.model'),
> get_json_object(json, '$[0].model')) AS model,
> coalesce(
> get_json_object(json, '$.request_id'),
> get_json_object(json, '$[0].request_id')) AS request_id
> FROM events
> {code}
> For an array-shaped row, each {{coalesce}} evaluates both extractions, and
> the parsing cost continues to scale with the number of projected fields.
> With {{spark.sql.optimizer.getJsonObjectSharedParsing.enabled=true}},
> nonnegative literal array-index paths such as {{$[0].model}},
> {{$.items[0].id}}, and {{$.matrix[0][1]}} should be eligible for the existing
> shared parser. Safe object-or-array {{coalesce}} fallbacks over the same JSON
> attribute should also share compatible object-root and array-root paths
> without making later legacy fallback branches eager.
> The follow-up should preserve the existing {{get_json_object}} behavior for
> duplicate keys and parents, nulls, missing indexes, malformed or trailing
> JSON, non-container intermediate values, raw strings, rendering failures, and
> ancestor/descendant prefix conflicts. In particular, a terminal array-index
> JSON null must retain the existing string {{"null"}} result, while an
> object-key JSON null remains SQL null.
> Dynamic paths, wildcards, negative, invalid, or overflowing indexes,
> excessively deep paths, guarded conditionals, and unsafe {{coalesce}} shapes
> should continue using independent legacy evaluation. This should reuse the
> existing internal, default-disabled configuration and should not add a new
> user-facing expression or configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
