This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 8785728d3742 [SPARK-60803][SQL] Fix auto-filled DSv2 generated columns 
reading a struct field before the store assignment cast
8785728d3742 is described below

commit 8785728d37425607858120023cae7d11d94e617c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 08:27:15 2026 -0800

    [SPARK-60803][SQL] Fix auto-filled DSv2 generated columns reading a struct 
field before the store assignment cast
    
    ### What changes were proposed in this pull request?
    
    This PR fixes `TableOutputResolver.resolveGenerationExprReferences` so that 
a multi-part reference in the generation expression of an auto-filled generated 
column of a Data Source V2 table, such as `s.f`, reads the value stored in the 
column `s`.
    
    The first name part is replaced with the stored column, and the remaining 
parts become `UnresolvedExtractValue`, which is what the parser produces for a 
field of an expression. Previously, only single-part references were replaced, 
so `s.f` resolved against the column of the query, before the store assignment 
cast.
    
    ### Why are the changes needed?
    
    A write computes the generated value from a value that is not the one it 
stores. The row silently stores a generated value that does not match its own 
column, and a later write of that row fails.
    
    ```sql
    CREATE TABLE t (id INT, s STRUCT<f: INT>, i INT,
      b INT GENERATED ALWAYS AS (s.f * 10), d INT GENERATED ALWAYS AS (i * 10)) 
USING foo;
    INSERT INTO t (id, s, i) SELECT 1, named_struct('f', 5.7D), 5.7D;
    -- stored: [1,[5],5,57,50], where b should be 50
    INSERT INTO t SELECT * FROM t;  -- CHECK_CONSTRAINT_VIOLATION
    ```
    
    `d` above and `s['f']` already read the stored value.
    
    The same gap affects these writes:
    
    - A field of a nested struct (`s.n.f`), a map key (`m.k`), and a field of 
an array of structs (`a.f`).
    - A stored value that differs without a cast. `concat(s.f, '|')` on a 
`CHAR(3)` field stored `a|` beside the stored `a  `.
    - A field the query does not provide. `INSERT WITH SCHEMA EVOLUTION` 
filling the missing field failed with `FIELD_NOT_FOUND` when 
`spark.sql.insertNestedTypeCoercion.enabled` is `true`.
    - A column filled with its default value. It failed with 
`UNRESOLVED_COLUMN` when 
`spark.sql.lateralColumnAlias.enableImplicitResolution` is `false`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. This fixes a correctness bug in a feature released in Spark 4.3.0. An 
auto-filled generated column now reads a field of another column from the 
stored value, so the query above stores `50` instead of `57`, and the writes 
that failed above now succeed. Rows that are already stored keep their values. 
The SQL migration guide describes this.
    
    ### How was this patch tested?
    
    Added a test to `GeneratedColumnWriteSuite`. Without this fix, it fails 
with `[[5],57]` instead of `[[5],50]`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60008 from dongjoon-hyun/SPARK-60803.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit cadd221225de27974bd1e401bec3417ce1d5cee9)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-migration-guide.md                               |  1 +
 .../spark/sql/catalyst/analysis/TableOutputResolver.scala |  7 +++++--
 .../spark/sql/connector/GeneratedColumnWriteSuite.scala   | 15 +++++++++++++++
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index f7c89757a8bc..3fffe38ef71f 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -28,6 +28,7 @@ license: |
 - Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic 
Python UDF filter predicates that reference only partition columns. Previously, 
these predicates could be dropped, returning rows that did not satisfy the 
filter (SPARK-60486). Data derived from affected query results may need to be 
recomputed.
 - Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`, 
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of 
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so 
existing paths can match different keys or become invalid. For example, use 
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes 
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set 
`spark.sql.legacy.variantPathB [...]
 
+- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table 
reads a field of another column, such as `s.f`, from the value stored in that 
column. Previously, it read the value of the query before the store assignment 
cast, so it could silently store a value that does not match the stored column, 
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION` 
(SPARK-60803). Rows that are already stored keep their values, and data derived 
from them may need to be re [...]
 - Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled` 
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed 
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before 
Spark 4.2. With columnar input, they kept the columns that they pass through 
until the results of their batch came back from Python, and failed or returned 
wrong results when the operator reused or released the vectors of the batch in 
the meantime. To enab [...]
 - Since Spark 4.4, the conversion of columnar batches to rows releases each 
batch with `ColumnVector.closeIfFreeable` once its rows are consumed also when 
it is not whole-stage code-generated, e.g. when the query has more than 
`spark.sql.codegen.maxFields` fields or `spark.sql.codegen.wholeStage` is 
`false`, as the generated code has done since Spark 3.5.4 and 4.0. A 
`ColumnVector` that a Data Source V2 source reuses across batches, such as an 
`ArrowColumnVector` over a `VectorSchemaRoot [...]
 - Since Spark 4.4, `spark.sql.charVarchar.standardSemantics.enabled` (default 
`false`) is a preview feature that may change in future releases. When true, it 
enables first-class `CHAR(n)` and `VARCHAR(n)` semantics throughout schemas and 
query plans: `CHAR` -> `VARCHAR` -> `STRING` form a type-precedence family, 
explicit casts can return constrained types, least-common-type resolution for 
`COALESCE`/`CASE`/`UNION` can return `CHAR` or `VARCHAR`, and string 
expressions that transform thei [...]
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
index 62a67c51681a..1f84bf7c47d4 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
@@ -832,9 +832,12 @@ object TableOutputResolver extends SQLConfHelper with 
Logging {
       matched: Seq[NamedExpression],
       conf: SQLConf): Expression = {
     genExpr.transform {
-      case u: UnresolvedAttribute if u.nameParts.size == 1 =>
+      case u @ UnresolvedAttribute(name +: nestedFields) =>
         matched.collectFirst {
-          case alias: Alias if conf.resolver(alias.name, u.nameParts.head) => 
alias.child
+          case alias: Alias if conf.resolver(alias.name, name) =>
+            nestedFields.foldLeft(alias.child) { (child, field) =>
+              UnresolvedExtractValue(child, Literal(field))
+            }
         }.getOrElse(u)
     }
   }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
index 274ec02f262d..339da736adaf 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
@@ -322,6 +322,21 @@ class GeneratedColumnWriteSuite extends QueryTest with 
DatasourceV2SQLBase {
     }
   }
 
+  test("SPARK-60803: INSERT by name computes generated column from the stored 
(post-cast) " +
+    "struct field") {
+    val tblName = "my_tab"
+    withTable(s"testcat.$tblName") {
+      sql(s"""CREATE TABLE testcat.$tblName(
+             |  s STRUCT<f: INT>,
+             |  g INT GENERATED ALWAYS AS (s.f * 10)
+             |) USING foo""".stripMargin)
+      // 5.7 is cast to INT (= 5) when stored in `s.f`. The generated column 
must be computed from
+      // the stored value (5 * 10 = 50), not from the pre-cast value (5.7 * 10 
= 57).
+      sql(s"INSERT INTO testcat.$tblName(s) SELECT named_struct('f', 5.7D)")
+      checkAnswer(spark.table(s"testcat.$tblName"), Row(Row(5), 50))
+    }
+  }
+
   test("INSERT by position validates generated column against the stored 
(post-cast) value") {
     val tblName = "my_tab"
     withTable(s"testcat.$tblName") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to