This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8da92112f673 [SPARK-60817][SQL] Fix auto-filled DSv2 generated columns
reading a column in a lambda function before the store assignment cast
8da92112f673 is described below
commit 8da92112f673d1e9d3f780857cc8e014c89a8a20
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 14:59:54 2026 -0800
[SPARK-60817][SQL] Fix auto-filled DSv2 generated columns reading a column
in a lambda function before the store assignment cast
### What changes were proposed in this pull request?
This PR fixes a correctness issue: an auto-filled generated column of a
Data Source V2 table read a column referenced in a lambda function, such as `i`
in `transform(a, x -> x + i)`, from the query before the store assignment cast,
so it could silently store a wrong value.
#60008 fixed the same issue for a field of another column, such as `s.f`.
This PR fixes it for a column referenced in a lambda function, which was found
in the review of #60008.
Previously, `TableOutputResolver.resolveGenerationExprReferences` replaced
the column references in the generation expression with the expressions of the
stored values. The parser turns every name in a lambda function into
`UnresolvedNamedLambdaVariable`, so the rewrite did not see a column referenced
in a lambda function, and `ResolveLambdaVariables` later turned it back into
`UnresolvedAttribute`, which resolved against the query.
This PR removes the rewrite. When a write auto-fills a generated column,
`TableOutputResolver` now projects the stored values first and puts the
generation expressions in a projection on top of it, so the analyzer resolves
them against the stored columns in the same way as any other expression. Each
stored value is computed once per row, also when a lambda function reads it.
### Why are the changes needed?
A write computes the generated value from a value that is not the one it
stores. The row silently stores a generated value that does not match its own
column, and a later write of that row fails. These are the three examples from
the review of #60008:
```sql
CREATE TABLE t (arr ARRAY<INT>, i INT,
g ARRAY<INT> GENERATED ALWAYS AS (transform(arr, x -> x * i))) USING foo;
INSERT INTO t (arr, i) SELECT array(10), 5.7D;
-- Issue 1: g should be [50], but it stores [57]
INSERT INTO t SELECT * FROM t;
-- Issue 2: this should succeed, but it fails with
CHECK_CONSTRAINT_VIOLATION
```
- `transform(arr, x -> x * s.f)` on `s STRUCT<f: INT>` stores `[57]`
instead of `[50]` for `INSERT INTO t (arr, s) SELECT array(10),
named_struct('f', 5.7D)`.
- `aggregate(arr, 0, (acc, x) -> acc + x * i)` on `i INT` fails with
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` for `INSERT INTO t (arr, i) SELECT
array(10), 5.7D`, because `i` is the `DOUBLE` of the query. It should store
`50`.
The same gap affects these writes:
- A stored value that differs without a cast. `transform(a, x -> concat(x,
c, '|'))` on `c CHAR(3)` stored `pa|` beside the stored `a `.
- `map_filter(m, (k, v) -> v > i * 10)` stored an empty map for `map('k',
52)` and `i` of `5.7D`, instead of `{k -> 52}`.
- A column filled with its default value. It failed with
`UNRESOLVED_COLUMN` when the column is declared after the generated column, or
when `spark.sql.lateralColumnAlias.enableImplicitResolution` is `false`.
The rewrite also missed a column referenced with `IDENTIFIER`.
`IDENTIFIER('i') * 10` stored `57` instead of `50` for `i` of `5.7D`.
### Does this PR introduce _any_ user-facing change?
Yes. This fixes a correctness bug in a feature released in Spark 4.3.0. An
auto-filled generated column now reads a column referenced in a lambda function
or with `IDENTIFIER` from the stored value, so the three examples above store
`[50]`, `[50]`, and `50`, `IDENTIFIER('i') * 10` stores `50`, and the writes
that failed above now succeed. Rows that are already stored keep their values.
The SQL migration guide describes this.
A write with `spark.sql.caseSensitive` set to `true` to a table whose
generation expression refers to a column in another case, such as `I * 10` for
a column `i`, now fails with `UNRESOLVED_COLUMN` instead of `INTERNAL_ERROR`.
### How was this patch tested?
Added three tests to `GeneratedColumnWriteSuite`. Without this fix, the
first fails with `[[1],5,[58],[[1]]]` instead of `[[1],5,[51],[[1]]]`, the
second, where a DEFAULT column is declared after the generated column, fails
with `UNRESOLVED_COLUMN`, and the third, with `IDENTIFIER('i') * 10`, fails
with `[5,57]` instead of `[5,50]`. The first test also checks that a lambda
variable with the name of a column, `i` in `transform(a, i -> transform(a, x ->
i))`, still reads the lambda variable.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60035 from dongjoon-hyun/SPARK-60817.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 2 +-
.../catalyst/analysis/TableOutputResolver.scala | 59 ++++++----------------
.../sql/connector/GeneratedColumnWriteSuite.scala | 46 +++++++++++++++++
3 files changed, 63 insertions(+), 44 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 92671a6d01f4..414618410c79 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -33,7 +33,7 @@ license: |
- Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic
Python UDF filter predicates that reference only partition columns. Previously,
these predicates could be dropped, returning rows that did not satisfy the
filter (SPARK-60486). Data derived from affected query results may need to be
recomputed.
- Since Spark 4.4, when a file source reads a partition column with a
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory
names, a value that cannot be cast to that type fails with
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously,
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such
as `2020-01-01 25:00:00`, became the start of that da [...]
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
-
+- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a column referenced in a lambda function, such as `i` in `transform(a, x
-> x + i)`, or with `IDENTIFIER`, such as `IDENTIFIER('i')`, from the value
stored in that column. Previously, it read the value of the query before the
store assignment cast, so it could silently store a value that does not match
the stored column, and a later write of that row failed with
`CHECK_CONSTRAINT_VIOLATION` (SPARK-60817). [...]
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a field of another column, such as `s.f`, from the value stored in that
column. Previously, it read the value of the query before the store assignment
cast, so it could silently store a value that does not match the stored column,
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION`
(SPARK-60803). Rows that are already stored keep their values, and data derived
from them may need to be re [...]
- Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before
Spark 4.2. With columnar input, they kept the columns that they pass through
until the results of their batch came back from Python, and failed or returned
wrong results when the operator reused or released the vectors of the batch in
the meantime. To enab [...]
- Since Spark 4.4, the conversion of columnar batches to rows releases each
batch with `ColumnVector.closeIfFreeable` once its rows are consumed also when
it is not whole-stage code-generated, e.g. when the query has more than
`spark.sql.codegen.maxFields` fields or `spark.sql.codegen.wholeStage` is
`false`, as the generated code has done since Spark 3.5.4 and 4.0. A
`ColumnVector` that a Data Source V2 source reuses across batches, such as an
`ArrowColumnVector` over a `VectorSchemaRoot [...]
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
index 1f84bf7c47d4..82f15532c542 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala
@@ -193,7 +193,15 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
tableName, expected.map(col => toSQLId(Seq(col.name))).mkString(", "))
}
- val plan = if (resolved == query.output) {
+ val plan = if (autoFilledGenCols.nonEmpty) {
+ // The analyzer resolves the generation expressions of the auto-filled
generated columns
+ // against a projection of the stored columns, after any store
assignment cast. Each stored
+ // value is computed once per row, also when a lambda function reads it.
+ val stored = Project(resolved.filterNot(col =>
autoFilledGenCols.contains(col.name)), query)
+ Project(resolved.map { col =>
+ if (autoFilledGenCols.contains(col.name)) col else col.toAttribute
+ }, stored)
+ } else if (resolved == query.output) {
query
} else {
Project(resolved, query)
@@ -422,10 +430,9 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
GeneratedColumn.getGenerationExpression(expectedCol.metadata) match {
case Some(genExprSql) =>
autoFilledGenCols += expectedCol.name
- // The parsed references here are placeholders: they are resolved
against the post-cast
- // stored columns after this loop (see below), so the generated
value is computed from
- // the values as they will be written. References to other
generated columns are not
- // allowed.
+ // The references are left unresolved here.
`resolveOutputColumnsInternal` resolves them
+ // against the stored columns, so the generated value is computed
from the values as
+ // they will be written. References to other generated columns are
not allowed.
val genExpr = CatalystSqlParser.parseExpression(genExprSql)
Some(applyColumnMetadata(genExpr, expectedCol))
case None =>
@@ -478,23 +485,6 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
}
}
- // Resolve auto-filled generated column references against the post-cast
stored columns, so the
- // generated value is computed from the values as they will be written
(after any
- // store-assignment cast). Generated columns only exist at the top level,
- // so this only applies when colPath is empty.
- val finalCols = if (colPath.isEmpty && autoFilledGenCols.nonEmpty) {
- val storedCols = reordered.filterNot(col =>
autoFilledGenCols.contains(col.name))
- reordered.map { col =>
- if (autoFilledGenCols.contains(col.name)) {
- resolveGenerationExprReferences(col, storedCols,
conf).asInstanceOf[NamedExpression]
- } else {
- col
- }
- }
- } else {
- reordered
- }
-
val output = if (reordered.length == expectedCols.length) {
if (matchedCols.size < inputCols.length) {
val extraCols = inputCols.filterNot(col =>
matchedCols.contains(col.name))
@@ -507,7 +497,7 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
tableName, colPath.quoted, extraCols)
}
} else {
- finalCols
+ reordered
}
} else if (enforceFullOutput) {
val colName =
@@ -600,12 +590,10 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
GeneratedColumn.getGenerationExpression(expectedCol.metadata) match {
case Some(genExprSql) =>
autoFilledGenCols += expectedCol.name
+ // The references are resolved as in the by-name path. The stored
columns have the
+ // names of the table columns, even when the query uses different
names.
val genExpr = CatalystSqlParser.parseExpression(genExprSql)
- // For by-position, manually resolve references against matched
columns
- // since the query may use different column names than the table.
References to
- // other generated columns are not allowed.
- val resolvedGenExpr = resolveGenerationExprReferences(genExpr,
matched, conf)
- applyColumnMetadata(resolvedGenExpr, expectedCol)
+ applyColumnMetadata(genExpr, expectedCol)
case None =>
val useNullAsDefault =
useNullAsDefaultForMissingColumn(actualExpectedCols, conf)
val defaultExpr = if (fillDefaultValue) {
@@ -827,21 +815,6 @@ object TableOutputResolver extends SQLConfHelper with
Logging {
!expectedCols.exists(c =>
GeneratedColumn.isGeneratedColumn(c.metadata)))
}
- private def resolveGenerationExprReferences(
- genExpr: Expression,
- matched: Seq[NamedExpression],
- conf: SQLConf): Expression = {
- genExpr.transform {
- case u @ UnresolvedAttribute(name +: nestedFields) =>
- matched.collectFirst {
- case alias: Alias if conf.resolver(alias.name, name) =>
- nestedFields.foldLeft(alias.child) { (child, field) =>
- UnresolvedExtractValue(child, Literal(field))
- }
- }.getOrElse(u)
- }
- }
-
// For table insertions, capture the overflow errors and show proper message.
// Without this method, the overflow errors of castings will show hints for
turning off ANSI SQL
// mode, which are not helpful since the behavior is controlled by the store
assignment policy.
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
index 339da736adaf..accba28d42da 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
@@ -337,6 +337,52 @@ class GeneratedColumnWriteSuite extends QueryTest with
DatasourceV2SQLBase {
}
}
+ test("SPARK-60817: INSERT by name computes generated column from the stored
(post-cast) " +
+ "value in a lambda") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a ARRAY<INT>,
+ | i INT,
+ | g1 ARRAY<INT> GENERATED ALWAYS AS (transform(a, x -> x + i *
10)),
+ | g2 ARRAY<ARRAY<INT>> GENERATED ALWAYS AS (transform(a, i ->
transform(a, x -> i)))
+ |) USING foo""".stripMargin)
+ // 5.7 is cast to INT (= 5) when stored in `i`. `g1` must read the
stored value (1 + 5 * 10),
+ // not the pre-cast value (1 + 5.7 * 10). In `g2`, `i` is the lambda
variable, not the column.
+ sql(s"INSERT INTO testcat.$tblName(a, i) SELECT array(1), 5.7D")
+ checkAnswer(spark.table(s"testcat.$tblName"), Row(Seq(1), 5, Seq(51),
Seq(Seq(1))))
+ }
+ }
+
+ test("SPARK-60817: INSERT by name computes generated column in a lambda from
a DEFAULT " +
+ "column declared after it") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | a ARRAY<INT>,
+ | g ARRAY<INT> GENERATED ALWAYS AS (transform(a, x -> x + d)),
+ | d INT DEFAULT 5
+ |) USING foo""".stripMargin)
+ // `d` is filled with its default value, which `g` must read (1 + 5).
+ sql(s"INSERT INTO testcat.$tblName(a) SELECT array(1)")
+ checkAnswer(spark.table(s"testcat.$tblName"), Row(Seq(1), Seq(6), 5))
+ }
+ }
+
+ test("SPARK-60817: INSERT by name computes generated column from the stored
(post-cast) " +
+ "value of a column referenced with IDENTIFIER") {
+ val tblName = "my_tab"
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | i INT,
+ | g INT GENERATED ALWAYS AS (IDENTIFIER('i') * 10)
+ |) USING foo""".stripMargin)
+ // 5.7 is cast to INT (= 5) when stored in `i`. `g` must read the stored
value (5 * 10).
+ sql(s"INSERT INTO testcat.$tblName(i) SELECT 5.7D")
+ checkAnswer(spark.table(s"testcat.$tblName"), Row(5, 50))
+ }
+ }
+
test("INSERT by position validates generated column against the stored
(post-cast) value") {
val tblName = "my_tab"
withTable(s"testcat.$tblName") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]