ganeshashree opened a new pull request, #57559:
URL: https://github.com/apache/spark/pull/57559
### What changes were proposed in this pull request?
Add the ANSI SQL:2016 `JSON_TABLE` table-valued function, which shreds a
JSON document into a relational table. A row path selects a sequence of JSON
items and a `COLUMNS` clause projects a typed value out of each item into a
column.
Syntax (flat, non-nested subset):
JSON_TABLE(json_expr, row_path
COLUMNS (
col1 FOR ORDINALITY,
col2 <type> [PATH '<json_path>'],
col3 <type> EXISTS [PATH '<json_path>']
)
[ { NULL | ERROR } ON ERROR ]
) [AS] alias
- A row path ending in `[*]` expands a JSON array into one row per element;
a non-wildcard path yields a single row for the matched value.
- `FOR ORDINALITY`: a 1-based BIGINT row counter.
- Value columns are extracted and cast to the declared type; the path may be
explicit (`PATH '...'`) or implicit (`'$.<columnName>'`).
- `EXISTS` columns are a presence test cast to the declared type; a
present-but-null JSON value counts as existing, only an absent path is false.
- `{ NULL | ERROR } ON ERROR`: `NULL ON ERROR` (default) yields no rows on
null/malformed input; `ERROR ON ERROR` raises.
- Usable in a comma join and with `LATERAL`.
Implementation notes:
- A `JsonTable` `Generator` expression is wrapped by the existing `Generate`
operator, so no new execution operator is introduced.
- A dedicated grammar production handles the `COLUMNS(...)` clause;
`AstBuilder.visitJsonTableRelation` builds the plan. `ERROR`, `ORDINALITY`, and
`JSON_TABLE` are added as non-reserved keywords.
- A token-aware navigator (`JsonTableEvaluator`) extracts values so that a
missing path, a JSON `null`, and the literal string `"null"` are all
distinguished (unlike `get_json_object`). Non-trailing wildcard paths are
rejected during analysis via `DATATYPE_MISMATCH.INVALID_JSON_TABLE_PATH`.
- Only the flat subset is implemented; `NESTED PATH`, `FORMAT JSON` query
columns, and `DEFAULT ... ON EMPTY` are left as follow-ups.
### Why are the changes needed?
`JSON_TABLE` is the SQL-standard way to turn JSON into rows and columns, and
is supported by Oracle, DB2, MySQL 8, PostgreSQL 17, Snowflake, and Trino.
Spark previously required chaining `from_json` + `explode`/`inline` +
`get_json_object` to achieve the same result. This folds that into one
declarative, standard construct and eases migration from those systems.
### Does this PR introduce _any_ user-facing change?
Yes. It adds the `JSON_TABLE` SQL table-valued function and a SQL reference
documentation page. `ERROR`, `ORDINALITY`, and `JSON_TABLE` are added as
non-reserved keywords, so existing queries using them as identifiers continue
to parse. There is no change to existing behavior.
### How was this patch tested?
New end-to-end suite `JsonTableSuite` covering array expansion, ordinality,
explicit/implicit paths, missing-vs-null semantics for value and EXISTS
columns, `NULL`/`ERROR ON ERROR`, `[*]` over a non-array, mid-path wildcard
rejection, non-explode string row items, arrays of scalars/strings, duplicate
keys, structure-valued columns, `LATERAL` joins, nested-field extraction, and
column aliases. Existing `JsonExpressionsSuite`, `JsonFunctionsSuite`,
`GeneratorFunctionSuite`, `PlanParserSuite`, `DDLParserSuite`,
`SQLKeywordSuite`, and `SparkThrowableSuite` continue to pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
