sunchao opened a new pull request, #57069: URL: https://github.com/apache/spark/pull/57069
## Why are the changes needed? [SPARK-47670](https://issues.apache.org/jira/browse/SPARK-47670) introduced opt-in shared parsing for repeated `get_json_object` calls over the same JSON input. [SPARK-57626](https://issues.apache.org/jira/browse/SPARK-57626) extended the optimization from top-level fields to repeated literal nested object paths. Array indexes and conditional expressions remain outside the shared-parsing path. As described in [SPARK-57990](https://issues.apache.org/jira/browse/SPARK-57990), this leaves a common object-or-array payload pattern parsing the same JSON repeatedly: ```sql SELECT coalesce( get_json_object(json, '$.model'), get_json_object(json, '$[0].model')) AS model, coalesce( get_json_object(json, '$.request_id'), get_json_object(json, '$[0].request_id')) AS request_id FROM events ``` For an array-shaped row, each `coalesce` evaluates both extractions. The parsing cost therefore continues to scale with the number of projected fields. ## What changes were proposed in this PR? This PR extends the existing internal shared parser and optimizer rewrite to array-index paths and safe object-or-array `coalesce` fallbacks. The changes: - Represent eligible literal JSON paths as named-key and nonnegative array-index segments. This supports paths such as `$.items[0].id`, `$[0].model`, and `$.matrix[0][1]`. - Extend the runtime path trie to traverse object and array roots, nested arrays, and mixed object/array paths in one streaming parse. - Share compatible `coalesce` branches when every branch is an eligible `get_json_object`, optionally wrapped in casts, and every branch reads the same JSON attribute. - Select only the first object-root and first array-root branch from each eligible `coalesce`. These root shapes are mutually exclusive, while later same-root fallbacks remain lazy. - Keep each selected object/array pair together as one atomic grouping unit when prefix conflicts require multiple shared-parser groups. - Preserve legacy behavior for duplicate keys and parents, nulls, missing indexes, malformed or trailing JSON, non-container intermediate values, raw strings, rendering failures, and ancestor/descendant prefix conflicts. Terminal array-index JSON null remains the string `"null"`, while object-key JSON null remains SQL null. - Leave dynamic paths, wildcards, negative, invalid, or overflowing indexes, paths deeper than 64 segments, guarded conditionals, and unsafe `coalesce` shapes on independent `get_json_object` evaluation. - Update the existing configuration description and add optimizer, runtime, code-generation, malformed-input, prefix-grouping, and benchmark coverage. This reuses the existing internal, default-disabled `spark.sql.optimizer.getJsonObjectSharedParsing.enabled` configuration. It does not add a new expression, configuration, or query migration. ## How was this PR tested? The following checks passed locally on JDK 17: ```bash build/sbt \ "catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizeJsonExprsSuite" build/sbt \ "sql/testOnly org.apache.spark.sql.JsonFunctionsSuite" build/sbt \ "catalyst/scalastyle" \ "catalyst/Test/scalastyle" \ "sql/scalastyle" \ "sql/Test/scalastyle" git diff --check ``` - `OptimizeJsonExprsSuite`: 31 passed. - `JsonFunctionsSuite`: 111 passed. - All four Scalastyle targets completed with zero errors and zero warnings. - `git diff --check` passed. The added coverage includes array and mixed object/array paths, safe and unsafe `coalesce` shapes, cast preservation, atomic prefix grouping, malformed input, duplicate keys and parents, JSON null behavior, missing indexes, unsupported paths, and whole-stage code generation. A microbenchmark was added for 200,000 cached rows, 32 fields, and a 50/50 mix of object and one-element-array payloads, projecting 2, 4, 8, and 16 object/array fallback fields. Spark's GitHub Actions benchmark workflow will generate the Linux result files for JDK 17, 21, and 25 before the PR is marked ready for review. ## Does this PR introduce any user-facing change? Yes, within the unreleased `master` branch only. When the existing internal shared-parsing configuration is enabled, eligible repeated array-index paths and object-or-array `coalesce` fallbacks now use one shared parse. Query results and public APIs are unchanged. With the configuration disabled, analyzed and optimized plans remain unchanged. Released Spark versions are unaffected. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
