This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 8785728d3742 [SPARK-60803][SQL] Fix auto-filled DSv2 generated columns
reading a struct field before the store assignment cast
8785728d3742 is described below
commit 8785728d37425607858120023cae7d11d94e617c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 08:27:15 2026 -0800
[SPARK-60803][SQL] Fix auto-filled DSv2 generated columns reading a struct
field before the store assignment cast
### What changes were proposed in this pull request?
This PR fixes `TableOutputResolver.resolveGenerationExprReferences` so that
a multi-part reference in the generation expression of an auto-filled generated
column of a Data Source V2 table, such as `s.f`, reads the value stored in the
column `s`.
The first name part is replaced with the stored column, and the remaining
parts become `UnresolvedExtractValue`, which is what the parser produces for a
field of an expression. Previously, only single-part references were replaced,
so `s.f` resolved against the column of the query, before the store assignment
cast.
### Why are the changes needed?
A write computes the generated value from a value that is not the one it
stores. The row silently stores a generated value that does not match its own
column, and a later write of that row fails.
```sql
CREATE TABLE t (id INT, s STRUCT<f: INT>, i INT,
b INT GENERATED ALWAYS AS (s.f * 10), d INT GENERATED ALWAYS AS (i * 10))
USING foo;
INSERT INTO t (id, s, i) SELECT 1, named_struct('f', 5.7D), 5.7D;
-- stored: [1,[5],5,57,50], where b should be 50
INSERT INTO t SELECT * FROM t; -- CHECK_CONSTRAINT_VIOLATION
```
`d` above and `s['f']` already read the stored value.
The same gap affects these writes:
- A field of a nested struct (`s.n.f`), a map key (`m.k`), and a field of
an array of structs (`a.f`).
- A stored value that differs without a cast. `concat(s.f, '|')` on a
`CHAR(3)` field stored `a|` beside the stored `a `.
- A field the query does not provide. `INSERT WITH SCHEMA EVOLUTION`
filling the missing field failed with `FIELD_NOT_FOUND` when
`spark.sql.insertNestedTypeCoercion.enabled` is `true`.
- A column filled with its default value. It failed with
`UNRESOLVED_COLUMN` when
`spark.sql.lateralColumnAlias.enableImplicitResolution` is `false`.
### Does this PR introduce _any_ user-facing change?
Yes. This fixes a correctness bug in a feature released in Spark 4.3.0. An
auto-filled generated column now reads a field of another column from the
stored value, so the query above stores `50` instead of `57`, and the writes
that failed above now succeed. Rows that are already stored keep their values.
The SQL migration guide describes this.
### How was this patch tested?
Added a test to `GeneratedColumnWriteSuite`. Without this fix, it fails
with `[[5],57]` instead of `[[5],50]`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60008 from dongjoon-hyun/SPARK-60803.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit cadd221225de27974bd1e401bec3417ce1d5cee9)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../spark/sql/catalyst/analysis/TableOutputResolver.scala | 7 +++++--
.../spark/sql/connector/GeneratedColumnWriteSuite.scala | 15 +++++++++++++++
3 files changed, 21 insertions(+), 2 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index f7c89757a8bc..3fffe38ef71f 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -28,6 +28,7 @@ license: |
- Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic
Python UDF filter predicates that reference only partition columns. Previously,
these predicates could be dropped, returning rows that did not satisfy the
filter (SPARK-60486). Data derived from affected query results may need to be
recomputed.
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
+- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a field of another column, such as `s.f`, from the value stored in that
column. Previously, it read the value of the query before the store assignment
cast, so it could silently store a value that does not match the stored column,
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION`
(SPARK-60803). Rows that are already stored keep their values, and data derived
from them may need to be re [...]
- Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before
Spark 4.2. With columnar input, they kept the columns that they pass through
until the results of their batch came back from Python, and failed or returned
wrong results when the operator reused or released the vectors of the batch in
the meantime. To enab [...]
- Since Spark 4.4, the conversion of columnar batches to rows releases each
batch with `ColumnVector.closeIfFreeable` once its rows are consumed also when
it is not whole-stage code-generated, e.g. when the query has more than
`spark.sql.codegen.maxFields` fields or `spark.sql.codegen.wholeStage` is
`false`, as the generated code has done since Spark 3.5.4 and 4.0. A
`ColumnVector` that a Data Source V2 source reuses across batches, such as an
`ArrowColumnVector` over a `VectorSchemaRoot [...]
- Since Spark 4.4, `spark.sql.charVarchar.standardSemantics.enabled` (default
`false`) is a preview feature that may change in future releases. When true, it
enables first-class `CHAR(n)` and `VARCHAR(n)` semantics throughout schemas and
query plans: `CHAR` -> `VARCHAR` -> `STRING` form a type-precedence family,
explicit casts can return constrained types, least-common-type resolution for
`COALESCE`/`CASE`/`UNION` can return `CHAR` or `VARCHAR`, and string
expressions that transform thei [...]
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
index 62a67c51681a..1f84bf7c47d4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
@@ -832,9 +832,12 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
matched: Seq[NamedExpression],
conf: SQLConf): Expression = {
genExpr.transform {
- case u: UnresolvedAttribute if u.nameParts.size == 1 =>
+ case u @ UnresolvedAttribute(name +: nestedFields) =>
matched.collectFirst {
- case alias: Alias if conf.resolver(alias.name, u.nameParts.head) =>
alias.child
+ case alias: Alias if conf.resolver(alias.name, name) =>
+ nestedFields.foldLeft(alias.child) { (child, field) =>
+ UnresolvedExtractValue(child, Literal(field))
+ }
}.getOrElse(u)
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
index 274ec02f262d..339da736adaf 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
@@ -322,6 +322,21 @@ class GeneratedColumnWriteSuite extends QueryTest with
DatasourceV2SQLBase {
}
}
+ test("SPARK-60803: INSERT by name computes generated column from the stored
(post-cast) " +
+ "struct field") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | s STRUCT<f: INT>,
+ | g INT GENERATED ALWAYS AS (s.f * 10)
+ |) USING foo""".stripMargin)
+ // 5.7 is cast to INT (= 5) when stored in `s.f`. The generated column
must be computed from
+ // the stored value (5 * 10 = 50), not from the pre-cast value (5.7 * 10
= 57).
+ sql(s"INSERT INTO testcat.$tblName(s) SELECT named_struct('f', 5.7D)")
+ checkAnswer(spark.table(s"testcat.$tblName"), Row(Row(5), 50))
+ }
+ }
+
test("INSERT by position validates generated column against the stored
(post-cast) value") {
val tblName = "my_tab"
withTable(s"testcat.$tblName") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]