This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 90fe4487876d [SPARK-57360][SQL] Block temporary variables in generated
column expressions
90fe4487876d is described below
commit 90fe4487876d927b43713f27430f59905eaca0bb
Author: Szehon Ho <[email protected]>
AuthorDate: Thu Jun 11 16:50:58 2026 -0700
[SPARK-57360][SQL] Block temporary variables in generated column expressions
### What changes were proposed in this pull request?
This PR adds a validation that rejects references to session (temporary)
variables in a generated column's generation expression, for both `CREATE
TABLE` and `REPLACE TABLE`. The check is added to the central
`GeneratedColumnExpression.validate` and throws
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` with a clear reason.
### Why are the changes needed?
This is a regression from #54126 ([SPARK-55347][SQL] Pass Generated Column
as Expression to DSV2). Previously, the generation expression was analyzed by
an independent (standalone) analyzer that did not resolve session variables, so
a generated column referencing a temporary variable was not accepted. After
#54126 moved the analysis inline into the main analyzer (similar to how
constraint expressions are analyzed), the analyzer's last-resort variable
resolution now resolves the refere [...]
Persisting a session-scoped, mutable value into a generation expression is
ill-defined: the value recomputed when reading or rewriting the column can
differ across sessions or over time. The SQL standard restricts generation
expressions to be deterministic and to reference only other columns of the same
row, and other engines explicitly disallow variables:
- **MySQL**: "Variables (system variables, user-defined variables, and
stored program local variables) are not permitted."
- **HSQLDB** (close to the standard): the expression "must reference only
other, non-generated, columns of the table in the same row ... must be
deterministic and must not access SQL-data."
This restores the previous protection and aligns Spark with the standard.
### Does this PR introduce _any_ user-facing change?
Yes. A `CREATE TABLE` / `REPLACE TABLE` whose generated column references a
temporary variable now fails analysis with
`UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` (reason: "generation expression
cannot reference temporary variables") instead of being accepted.
Example:
```sql
DECLARE my_var INT DEFAULT 1;
CREATE TABLE t(a INT, b INT GENERATED ALWAYS AS (a + my_var)) USING foo;
-- now fails with UNSUPPORTED_EXPRESSION_GENERATED_COLUMN
```
### How was this patch tested?
Added unit tests in `DataSourceV2SQLSuite` covering both `CREATE TABLE` and
`REPLACE TABLE`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56421 from szehon-ho/block_temp_variable_from_generated_column.
Authored-by: Szehon Ho <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
(cherry picked from commit 334952601082732e809ec12cf11319a60dd32ea4)
Signed-off-by: Szehon Ho <[email protected]>
---
.../plans/logical/GeneratedColumnExpression.scala | 10 ++++++++-
.../spark/sql/connector/DataSourceV2SQLSuite.scala | 24 ++++++++++++++++++++++
2 files changed, 33 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
index 28548dacdfbd..6253ccccafc3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.catalyst.plans.logical
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.expressions.{AnalysisAwareExpression,
AttributeReference, Cast, Expression, UnaryExpression, Unevaluable}
-import
org.apache.spark.sql.catalyst.trees.TreePattern.{ANALYSIS_AWARE_EXPRESSION,
PLAN_EXPRESSION, TreePattern}
+import
org.apache.spark.sql.catalyst.trees.TreePattern.{ANALYSIS_AWARE_EXPRESSION,
PLAN_EXPRESSION, TreePattern, VARIABLE_REFERENCE}
import org.apache.spark.sql.catalyst.util.V2ExpressionBuilder
import org.apache.spark.sql.connector.catalog.GenerationExpression
import org.apache.spark.sql.internal.SQLConf
@@ -57,6 +57,7 @@ case class GeneratedColumnExpression(
* - The expression must be deterministic
* - The expression data type can be safely up-cast to the destination
column data type
* - No subquery expressions
+ * - No references to session (temporary) variables
* - No non-UTF8 binary collation
*/
def validate(
@@ -77,6 +78,13 @@ case class GeneratedColumnExpression(
throw unsupportedExpressionError("subquery expressions are not allowed
for generated columns")
}
+ // Don't allow references to session (temporary) variables. They are
session-scoped mutable
+ // state, so persisting them into a generation expression would be
ill-defined.
+ if (child.containsPattern(VARIABLE_REFERENCE)) {
+ throw unsupportedExpressionError(
+ "generation expression cannot reference temporary variables")
+ }
+
// Use the resolver to respect case sensitivity settings
val resolver = SQLConf.get.resolver
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
index a5480d579a59..752c8ed6c22e 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
@@ -2020,6 +2020,30 @@ class DataSourceV2SQLSuiteV1Filter
}
}
+ test("SPARK-57360: generation expression cannot reference temporary
variables") {
+ val tblName = "my_tab"
+ withSessionVariable("my_var") {
+ sql("DECLARE OR REPLACE VARIABLE my_var INT DEFAULT 1")
+ checkUnsupportedGenerationExpressionForCreate(
+ tblName,
+ "a + my_var",
+ "generation expression cannot reference temporary variables"
+ )
+ }
+ }
+
+ test("SPARK-57360: REPLACE TABLE - generation expression cannot reference
temporary variables") {
+ val tblName = "my_tab"
+ withSessionVariable("my_var") {
+ sql("DECLARE OR REPLACE VARIABLE my_var INT DEFAULT 1")
+ checkUnsupportedGenerationExpressionForReplace(
+ tblName,
+ "a + my_var",
+ "generation expression cannot reference temporary variables"
+ )
+ }
+ }
+
test("SPARK-44313: generation expression validation passes when there is a
char/varchar column") {
val tblName = "my_tab"
// InMemoryTableCatalog.capabilities() =
{SUPPORTS_CREATE_TABLE_WITH_GENERATED_COLUMNS}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
