This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new da21add5f1f6 [SPARK-58373][SQL] Do not prune the from_json schema
under named_struct when parse options are set
da21add5f1f6 is described below
commit da21add5f1f6039e65795bed57e1c64f3b90027c
Author: YangJie <[email protected]>
AuthorDate: Thu Jul 30 14:27:23 2026 +0800
[SPARK-58373][SQL] Do not prune the from_json schema under named_struct
when parse options are set
### What changes were proposed in this pull request?
The `CreateNamedStruct` branch of `OptimizeCsvJsonExprs` prunes a
`from_json` schema down to the fields the struct selects, but unlike the
sibling `GetStructField` and `GetArrayStructFields` branches it does not
require the parse options to be empty. This adds the same `options.isEmpty`
guard, so the rewrite is skipped whenever any option is set.
### Why are the changes needed?
Pruning the schema stops the parser from converting the dropped fields, so
a malformed value in one of them is never reported. Under `mode=FAILFAST` that
turns a query that should fail into one that silently returns a row:
```sql
SELECT named_struct(
'a', from_json(value, 'a int, b int, c int', map('mode', 'FAILFAST')).a,
'b', from_json(value, 'a int, b int, c int', map('mode', 'FAILFAST')).b)
FROM data -- value = '{"a": 1, "b": 2, "c": "bad"}'
```
With `spark.sql.optimizer.enableJsonExpressionOptimization=false` this
raises `MALFORMED_RECORD_IN_PARSING`, because `c` is parsed and rejected. With
the optimization on (the default) the schema is pruned to `a int, b int`, `c`
is skipped, and the query returns `{a: 1, b: 2}`.
SPARK-32968 added this branch and SPARK-33907 added the `options.isEmpty`
guard the following day, but only to the two `GetStructField`-style branches,
so this one has been unguarded since 3.1.0.
### Does this PR introduce _any_ user-facing change?
Yes. `named_struct` over a `from_json` that carries parse options now
honors those options again: a malformed record fails under `FAILFAST` instead
of being silently accepted. Queries whose `from_json` has no options are
unaffected. Note the rewrite also collapses several `from_json` evaluations
into one; that consolidation is now skipped for the options-set case as well,
matching what SPARK-33907 already accepted for the sibling branches.
### How was this patch tested?
Added an end-to-end `checkError` test in `JsonFunctionsSuite` that runs the
query above with the optimization both on and off and asserts
`MALFORMED_RECORD_IN_PARSING` either way; it fails on the unfixed tree because
no exception is thrown. The bad field is a type mismatch rather than a
structurally broken record: a structural malformation fails at tokenization no
matter which schema is requested and would hide the pruning.
Added a plan-level test in `OptimizeJsonExprsSuite` asserting the rewrite
is skipped for two different option maps (a parse mode and a formatting option,
since the guard rejects any option), with an empty-options control asserting
the same shape is still rewritten.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #57605 from LuciferYang/SPARK-58373-json-prune-options.
Authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
(cherry picked from commit 92ad0991c1c9aaee995584bb93b7766d57b62dee)
Signed-off-by: yangjie01 <[email protected]>
---
.../catalyst/optimizer/OptimizeCsvJsonExprs.scala | 6 +++++-
.../optimizer/OptimizeJsonExprsSuite.scala | 18 ++++++++++++++++
.../org/apache/spark/sql/JsonFunctionsSuite.scala | 25 ++++++++++++++++++++++
3 files changed, 48 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
index ec4ac257b304..4f6c3a2da457 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
@@ -393,10 +393,14 @@ object OptimizeCsvJsonExprs extends Rule[LogicalPlan] {
private val jsonOptimization: PartialFunction[Expression, Expression] = {
case c: CreateNamedStruct
- // If we create struct from various fields of the same `JsonToStructs`.
+ // If we create struct from various fields of the same
`JsonToStructs`. Pruning the
+ // schema stops the parser from converting the dropped fields, so a
malformed value in
+ // one of them no longer fails under a parse mode such as failfast. To
be more
+ // conservative, it does not optimize when any option is set, like the
cases below.
if c.valExprs.forall { v =>
v.isInstanceOf[GetStructField] &&
v.asInstanceOf[GetStructField].child.isInstanceOf[JsonToStructs] &&
+
v.asInstanceOf[GetStructField].child.asInstanceOf[JsonToStructs].options.isEmpty
&&
v.children.head.semanticEquals(c.valExprs.head.children.head)
} =>
val jsonToStructs = c.valExprs.map(_.children.head)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
index dd0f3dd01327..96f46344ffda 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
@@ -690,6 +690,24 @@ class OptimizeJsonExprsSuite extends PlanTest with
ExpressionEvalHelper {
comparePlans(optimized2, query2.analyze)
}
+ test("SPARK-58373: do not simplify named_struct + from_json if options is
not empty") {
+ val schema = StructType.fromDDL("a int, b int, c long")
+ def query(options: Map[String, String]): LogicalPlan =
testRelation2.select(namedStruct(
+ "a", GetStructField(JsonToStructs(schema, options, $"json"), 0),
+ "b", GetStructField(JsonToStructs(schema, options, $"json"),
1)).as("struct")).analyze
+
+ // Any option disables the rewrite, not just a parse mode: pruning `c` out
of the schema
+ // stops the parser from converting it, which a caller may be relying on.
+ Seq(Map("mode" -> "failfast"), Map("timestampFormat" -> "yyyy")).foreach {
options =>
+ comparePlans(Optimizer.execute(query(options)), query(options))
+ }
+
+ // Control: the same shape with no options is still rewritten, so the
guard above is what
+ // blocks it rather than some other precondition of the rule.
+ val optimized = Optimizer.execute(query(Map.empty))
+ assert(optimized != query(Map.empty), "expected the empty-options plan to
be rewritten")
+ }
+
test("SPARK-33007: simplify named_struct + from_json") {
val options = Map.empty[String, String]
val schema = StructType.fromDDL("a int, b int, c long, d string")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
index 4bbd3b533d34..6eccbb7a6a38 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
@@ -1815,6 +1815,31 @@ class JsonFunctionsSuite extends SharedSparkSession {
}
}
+ test("SPARK-58373: bad json input with json pruning optimization:
named_struct") {
+ Seq("true", "false").foreach { enabled =>
+ withSQLConf(SQLConf.JSON_EXPRESSION_OPTIMIZATION.key -> enabled) {
+ // `c` is a type mismatch rather than a structural malformation, so it
only fails
+ // when the parser is actually asked for it. A structurally broken
record would
+ // fail at tokenization regardless of the requested schema and hide
the pruning.
+ // The two from_json calls are written inline: an aliased subquery
reference would
+ // not be inlined by CollapseProject, so the named_struct would not
see them.
+ val fromJson = "from_json(value, 'a int, b int, c int', map('mode',
'FAILFAST'))"
+ val df = Seq("""{"a": 1, "b": 2, "c": "bad"}""").toDS()
+ .selectExpr(s"named_struct('a', $fromJson.a, 'b', $fromJson.b)")
+
+ checkError(
+ exception = intercept[SparkException] {
+ df.collect()
+ },
+ condition = "MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION",
+ parameters = Map(
+ "badRecord" -> "[1,2,null]",
+ "failFastMode" -> "FAILFAST")
+ )
+ }
+ }
+ }
+
test("SPARK-47670: separately projected from_json fields preserve
malformed-input semantics") {
withSQLConf(
SQLConf.JSON_EXPRESSION_OPTIMIZATION.key -> "true",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]