This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new da21add5f1f6 [SPARK-58373][SQL] Do not prune the from_json schema 
under named_struct when parse options are set
da21add5f1f6 is described below

commit da21add5f1f6039e65795bed57e1c64f3b90027c
Author: YangJie <[email protected]>
AuthorDate: Thu Jul 30 14:27:23 2026 +0800

    [SPARK-58373][SQL] Do not prune the from_json schema under named_struct 
when parse options are set
    
    ### What changes were proposed in this pull request?
    
    The `CreateNamedStruct` branch of `OptimizeCsvJsonExprs` prunes a 
`from_json` schema down to the fields the struct selects, but unlike the 
sibling `GetStructField` and `GetArrayStructFields` branches it does not 
require the parse options to be empty. This adds the same `options.isEmpty` 
guard, so the rewrite is skipped whenever any option is set.
    
    ### Why are the changes needed?
    
    Pruning the schema stops the parser from converting the dropped fields, so 
a malformed value in one of them is never reported. Under `mode=FAILFAST` that 
turns a query that should fail into one that silently returns a row:
    
    ```sql
    SELECT named_struct(
      'a', from_json(value, 'a int, b int, c int', map('mode', 'FAILFAST')).a,
      'b', from_json(value, 'a int, b int, c int', map('mode', 'FAILFAST')).b)
    FROM data     -- value = '{"a": 1, "b": 2, "c": "bad"}'
    ```
    
    With `spark.sql.optimizer.enableJsonExpressionOptimization=false` this 
raises `MALFORMED_RECORD_IN_PARSING`, because `c` is parsed and rejected. With 
the optimization on (the default) the schema is pruned to `a int, b int`, `c` 
is skipped, and the query returns `{a: 1, b: 2}`.
    
    SPARK-32968 added this branch and SPARK-33907 added the `options.isEmpty` 
guard the following day, but only to the two `GetStructField`-style branches, 
so this one has been unguarded since 3.1.0.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `named_struct` over a `from_json` that carries parse options now 
honors those options again: a malformed record fails under `FAILFAST` instead 
of being silently accepted. Queries whose `from_json` has no options are 
unaffected. Note the rewrite also collapses several `from_json` evaluations 
into one; that consolidation is now skipped for the options-set case as well, 
matching what SPARK-33907 already accepted for the sibling branches.
    
    ### How was this patch tested?
    
    Added an end-to-end `checkError` test in `JsonFunctionsSuite` that runs the 
query above with the optimization both on and off and asserts 
`MALFORMED_RECORD_IN_PARSING` either way; it fails on the unfixed tree because 
no exception is thrown. The bad field is a type mismatch rather than a 
structurally broken record: a structural malformation fails at tokenization no 
matter which schema is requested and would hide the pruning.
    
    Added a plan-level test in `OptimizeJsonExprsSuite` asserting the rewrite 
is skipped for two different option maps (a parse mode and a formatting option, 
since the guard rejects any option), with an empty-options control asserting 
the same shape is still rewritten.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 5)
    
    Closes #57605 from LuciferYang/SPARK-58373-json-prune-options.
    
    Authored-by: YangJie <[email protected]>
    Signed-off-by: yangjie01 <[email protected]>
    (cherry picked from commit 92ad0991c1c9aaee995584bb93b7766d57b62dee)
    Signed-off-by: yangjie01 <[email protected]>
---
 .../catalyst/optimizer/OptimizeCsvJsonExprs.scala  |  6 +++++-
 .../optimizer/OptimizeJsonExprsSuite.scala         | 18 ++++++++++++++++
 .../org/apache/spark/sql/JsonFunctionsSuite.scala  | 25 ++++++++++++++++++++++
 3 files changed, 48 insertions(+), 1 deletion(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
index ec4ac257b304..4f6c3a2da457 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
@@ -393,10 +393,14 @@ object OptimizeCsvJsonExprs extends Rule[LogicalPlan] {
 
   private val jsonOptimization: PartialFunction[Expression, Expression] = {
     case c: CreateNamedStruct
-        // If we create struct from various fields of the same `JsonToStructs`.
+        // If we create struct from various fields of the same 
`JsonToStructs`. Pruning the
+        // schema stops the parser from converting the dropped fields, so a 
malformed value in
+        // one of them no longer fails under a parse mode such as failfast. To 
be more
+        // conservative, it does not optimize when any option is set, like the 
cases below.
         if c.valExprs.forall { v =>
           v.isInstanceOf[GetStructField] &&
             v.asInstanceOf[GetStructField].child.isInstanceOf[JsonToStructs] &&
+            
v.asInstanceOf[GetStructField].child.asInstanceOf[JsonToStructs].options.isEmpty
 &&
             v.children.head.semanticEquals(c.valExprs.head.children.head)
         } =>
       val jsonToStructs = c.valExprs.map(_.children.head)
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
index dd0f3dd01327..96f46344ffda 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJsonExprsSuite.scala
@@ -690,6 +690,24 @@ class OptimizeJsonExprsSuite extends PlanTest with 
ExpressionEvalHelper {
     comparePlans(optimized2, query2.analyze)
   }
 
+  test("SPARK-58373: do not simplify named_struct + from_json if options is 
not empty") {
+    val schema = StructType.fromDDL("a int, b int, c long")
+    def query(options: Map[String, String]): LogicalPlan = 
testRelation2.select(namedStruct(
+      "a", GetStructField(JsonToStructs(schema, options, $"json"), 0),
+      "b", GetStructField(JsonToStructs(schema, options, $"json"), 
1)).as("struct")).analyze
+
+    // Any option disables the rewrite, not just a parse mode: pruning `c` out 
of the schema
+    // stops the parser from converting it, which a caller may be relying on.
+    Seq(Map("mode" -> "failfast"), Map("timestampFormat" -> "yyyy")).foreach { 
options =>
+      comparePlans(Optimizer.execute(query(options)), query(options))
+    }
+
+    // Control: the same shape with no options is still rewritten, so the 
guard above is what
+    // blocks it rather than some other precondition of the rule.
+    val optimized = Optimizer.execute(query(Map.empty))
+    assert(optimized != query(Map.empty), "expected the empty-options plan to 
be rewritten")
+  }
+
   test("SPARK-33007: simplify named_struct + from_json") {
     val options = Map.empty[String, String]
     val schema = StructType.fromDDL("a int, b int, c long, d string")
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
index 4bbd3b533d34..6eccbb7a6a38 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
@@ -1815,6 +1815,31 @@ class JsonFunctionsSuite extends SharedSparkSession {
     }
   }
 
+  test("SPARK-58373: bad json input with json pruning optimization: 
named_struct") {
+    Seq("true", "false").foreach { enabled =>
+      withSQLConf(SQLConf.JSON_EXPRESSION_OPTIMIZATION.key -> enabled) {
+        // `c` is a type mismatch rather than a structural malformation, so it 
only fails
+        // when the parser is actually asked for it. A structurally broken 
record would
+        // fail at tokenization regardless of the requested schema and hide 
the pruning.
+        // The two from_json calls are written inline: an aliased subquery 
reference would
+        // not be inlined by CollapseProject, so the named_struct would not 
see them.
+        val fromJson = "from_json(value, 'a int, b int, c int', map('mode', 
'FAILFAST'))"
+        val df = Seq("""{"a": 1, "b": 2, "c": "bad"}""").toDS()
+          .selectExpr(s"named_struct('a', $fromJson.a, 'b', $fromJson.b)")
+
+        checkError(
+          exception = intercept[SparkException] {
+            df.collect()
+          },
+          condition = "MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION",
+          parameters = Map(
+            "badRecord" -> "[1,2,null]",
+            "failFastMode" -> "FAILFAST")
+        )
+      }
+    }
+  }
+
   test("SPARK-47670: separately projected from_json fields preserve 
malformed-input semantics") {
     withSQLConf(
         SQLConf.JSON_EXPRESSION_OPTIMIZATION.key -> "true",


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to