LuciferYang opened a new pull request, #57924:
URL: https://github.com/apache/spark/pull/57924
### What changes were proposed in this pull request?
`OptimizeCsvJsonExprs` prunes a `JsonToStructs` schema to the fields being
selected. Dropping a field stops the parser from converting it, so a malformed
value there is never detected and the corrupt record column comes back null
instead of the record text. This PR stops the two JSON branches from pruning in
that case.
The two branches need different conditions:
- The `GetStructField` branch prunes to exactly one field, so selecting the
corrupt record column there always drops the rest. It gets the same guard the
CSV branch added by SPARK-32968 already has.
- The `CreateNamedStruct` branch keeps every selected field, so it only
becomes unsafe when the selection is a strict subset of the schema. Guarding on
"corrupt column is selected" alone would give up the parse-deduplication
rewrite for an all-fields selection that drops nothing, turning one parse into
N.
`GetArrayStructFields` needs no guard: an `ArrayType` schema takes
`JsonToStructsEvaluator`'s `case other =>`, where `corruptFieldIndex` is
`None`, so corrupt-record semantics do not exist on that path.
### Why are the changes needed?
The corrupt record column silently returns null for a record that did fail
to parse. The rule requires `options.isEmpty`, which is the PERMISSIVE default,
and `spark.sql.optimizer.enableJsonExpressionOptimization` defaults to true, so
no configuration is needed to hit it:
```scala
Seq("""{"a": 1, "b": "bad"}""").toDS()
.selectExpr("from_json(value, 'a int, b int, _corrupt_record string') as
p")
.selectExpr("p._corrupt_record")
```
| spark.sql.optimizer.enableJsonExpressionOptimization | result |
|---|---|
| true | `null` |
| false | `{"a": 1, "b": "bad"}` |
The optimized plan shows the schema pruned to the corrupt column alone,
which leaves `actualSchema` empty so the parser has nothing to convert and
never raises `BadRecordException`:
```
Project [from_json(StructField(_corrupt_record,StringType,true), value#1,
...)._corrupt_record ...]
```
A type mismatch on a dropped field is needed to expose this. The existing
SPARK-33907 test uses a structurally malformed record, which fails at
tokenization whatever schema is requested, so it never depended on the pruning;
that test is now annotated to say the rule no longer fires for it.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes the wrong result above. With this PR the corrupt record column
holds the record text under both settings of the config, matching the
unoptimized behavior.
### How was this patch tested?
New tests, plan-level and end-to-end:
- `OptimizeJsonExprsSuite`: two `comparePlans` cases, each with a control.
One asserts the `GetStructField` rewrite is skipped for the corrupt column
while a normal field is still pruned; the other asserts the `CreateNamedStruct`
rewrite still fires when every field is selected and is skipped when the
selection drops a field.
- `JsonFunctionsSuite`: two `checkAnswer` cases covering both branches with
the optimization on and off.
Verified by reverting each guard on the fixed tree: removing the
`GetStructField` guard fails both new catalyst tests, and weakening
`prunesCorruptRecord` to "corrupt column selected" fails the `named_struct`
case. Full `JsonFunctionsSuite` (115) and `OptimizeJsonExprsSuite` +
`OptimizeCsvExprsSuite` (36) pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
