LuciferYang opened a new pull request, #57605:
URL: https://github.com/apache/spark/pull/57605
### What changes were proposed in this pull request?
The `CreateNamedStruct` branch of `OptimizeCsvJsonExprs` prunes a
`from_json` schema down to the fields the struct selects, but unlike the
sibling `GetStructField` and `GetArrayStructFields` branches it does not
require the parse options to be empty. This adds the same `options.isEmpty`
guard, so the rewrite is skipped whenever any option is set.
### Why are the changes needed?
Pruning the schema stops the parser from converting the dropped fields, so a
malformed value in one of them is never reported. Under `mode=FAILFAST` that
turns a query that should fail into one that silently returns a row:
```sql
SELECT named_struct(
'a', from_json(value, 'a int, b int, c int', map('mode', 'FAILFAST')).a,
'b', from_json(value, 'a int, b int, c int', map('mode', 'FAILFAST')).b)
FROM data -- value = '{"a": 1, "b": 2, "c": "bad"}'
```
With `spark.sql.optimizer.enableJsonExpressionOptimization=false` this
raises `MALFORMED_RECORD_IN_PARSING`, because `c` is parsed and rejected. With
the optimization on (the default) the schema is pruned to `a int, b int`, `c`
is skipped, and the query returns `{a: 1, b: 2}`.
SPARK-32968 added this branch and SPARK-33907 added the `options.isEmpty`
guard the following day, but only to the two `GetStructField`-style branches,
so this one has been unguarded since 3.1.0.
### Does this PR introduce _any_ user-facing change?
Yes. `named_struct` over a `from_json` that carries parse options now honors
those options again: a malformed record fails under `FAILFAST` instead of being
silently accepted. Queries whose `from_json` has no options are unaffected.
Note the rewrite also collapses several `from_json` evaluations into one; that
consolidation is now skipped for the options-set case as well, matching what
SPARK-33907 already accepted for the sibling branches.
### How was this patch tested?
Added an end-to-end `checkError` test in `JsonFunctionsSuite` that runs the
query above with the optimization both on and off and asserts
`MALFORMED_RECORD_IN_PARSING` either way; it fails on the unfixed tree because
no exception is thrown. The bad field is a type mismatch rather than a
structurally broken record: a structural malformation fails at tokenization no
matter which schema is requested and would hide the pruning.
Added a plan-level test in `OptimizeJsonExprsSuite` asserting the rewrite is
skipped for two different option maps (a parse mode and a formatting option,
since the guard rejects any option), with an empty-options control asserting
the same shape is still rewritten.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
