This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 33ebee39ef34 [SPARK-60821][SQL] Fix DSv2 CREATE/REPLACE TABLE
accepting a CHECK constraint that calls a generator function
33ebee39ef34 is described below
commit 33ebee39ef340ad74a58bb2d170ade0b877e8604
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 14:50:46 2026 -0800
[SPARK-60821][SQL] Fix DSv2 CREATE/REPLACE TABLE accepting a CHECK
constraint that calls a generator function
### What changes were proposed in this pull request?
This PR makes `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2 table
reject a CHECK constraint whose condition calls a generator function, such as
`explode`, `posexplode`, and `inline`. They fail with the new error condition
`GENERATOR_IN_CHECK_CONSTRAINT`, and the query context of the error points at
the call.
1. `ResolveTableSpec` looks for a `Generator` in the resolved condition. It
does this after the check for aggregate and window functions and before
`CheckConstraint.failIfCallsUserDefinedFunction`, so a Hive UDTF fails like a
built-in generator function.
2. The SQL migration guide describes this change.
`ALTER TABLE ... ADD CONSTRAINT` already rejects these conditions, and this
PR does not change its error. The check is in the DDL only, so a write to a
table with an existing constraint fails as before.
The error of `ALTER TABLE` is not reused because it shows the internal plan
instead of the condition:
```
[UNSUPPORTED_GENERATOR.OUTSIDE_SELECT] The generator is not supported:
outside the SELECT clause, found: Filter NOT isnotnull(explode(array(id#12))).
SQLSTATE: 42K0E
```
`AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` is not reused either,
because an aggregate or window function reads multiple rows, while a generator
function returns multiple rows.
### Why are the changes needed?
`CREATE TABLE` and `REPLACE TABLE` accept a generator function in a CHECK
constraint, but no write to the table can succeed after that:
```sql
CREATE TABLE cat.ns.t (id INT, CONSTRAINT c1 CHECK (explode(array(id)) IS
NOT NULL)) USING foo; -- succeeds
INSERT INTO cat.ns.t VALUES (1); -- UNSUPPORTED_GENERATOR.OUTSIDE_SELECT
```
https://github.com/apache/spark/pull/60005 rejected aggregate and window
functions in these statements, but not generator functions.
### Does this PR introduce _any_ user-facing change?
Yes.
- Before, `CREATE TABLE` and `REPLACE TABLE` accepted these conditions, and
every later write to the table failed. After, the statements fail:
```
[GENERATOR_IN_CHECK_CONSTRAINT] The check constraint `explode(array(id))
IS NOT NULL` calls a generator function. Check constraints compute one value
for each row and cannot call functions that return multiple rows. SQLSTATE:
42621
```
- Compared to the unreleased master branch, a Hive UDTF in `CREATE TABLE`
and `REPLACE TABLE` fails with this error instead of
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT`.
- `CREATE TABLE` on a V1 table with such a condition fails with this error
instead of `UNSUPPORTED_FEATURE.TABLE_OPERATION`.
Existing constraints that call a generator function are not rejected, and a
write to their tables keeps failing with the same error. Released Spark
versions accept these conditions since Apache Spark 4.1.0.
### How was this patch tested?
Pass the CIs with the newly added test case in `CheckConstraintSuite`. It
covers `CREATE TABLE` and `REPLACE TABLE` for `explode`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60039 from dongjoon-hyun/SPARK-60821.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit d4b33cad361dfa0c6c2f2ea95bac4fe8fe091a73)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 6 ++++++
docs/sql-migration-guide.md | 1 +
.../sql/catalyst/analysis/ResolveTableSpec.scala | 6 ++++++
.../execution/command/v2/CheckConstraintSuite.scala | 19 +++++++++++++++++++
4 files changed, 32 insertions(+)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 0a490786a155..127be3eacb5a 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -3102,6 +3102,12 @@
],
"sqlState" : "42623"
},
+ "GENERATOR_IN_CHECK_CONSTRAINT" : {
+ "message" : [
+ "The check constraint `<checkCondition>` calls a generator function.
Check constraints compute one value for each row and cannot call functions that
return multiple rows."
+ ],
+ "sqlState" : "42621"
+ },
"GEO_ENCODER_SRID_MISMATCH_ERROR" : {
"message" : [
"Failed to encode <type> value because provided SRID <valueSrid> of a
value to encode does not match type SRID: <typeSrid>."
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 0c46f88db266..50a4ec4244be 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -69,6 +69,7 @@ license: |
- Since Spark 4.4, `UPDATE` and `MERGE` on a Data Source V2 table fail with
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` when they assign to
a metadata column of the table. This includes `UPDATE SET *` and `INSERT *` in
`MERGE WITH SCHEMA EVOLUTION` when the source has a column with the name of a
metadata column of the target table. Previously, these assignments passed
analysis and were usually ignored. To restore the previous behavior, set
`spark.sql.legacy.ignoreMetadata [...]
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a user-defined scalar function, including a Data Source V2
function. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SQL_UDF_USAGE` on a SQL function. Previously, a write could
resolve the function name in its own session, so that it enforced another rule
or [...]
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` when the
condition of a CHECK constraint calls an aggregate or window function,
including a user-defined aggregate function, `grouping`, and `grouping_id`.
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with its existing errors, such
as `INVALID_WHERE_CONDITION` and `WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE`.
Previously, these statements succeeded and [...]
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `GENERATOR_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a generator function, such as `explode`, `posexplode`, and
`inline`. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_GENERATOR.OUTSIDE_SELECT`. Previously, these statements succeeded
and every later write to the table failed (SPARK-60821). To run these
statements, remove the function from the condition. E [...]
- Since Spark 4.4, `ALTER TABLE ... ADD CONSTRAINT` with a CHECK constraint
fails with `EXPECT_TABLE_NOT_VIEW.NO_ALTERNATIVE` when the target is a view, as
it does with the other constraint types. Previously, when the view only
projected or filtered a Data Source V2 table, the statement added the
constraint to that table after validating only the rows of the view
(SPARK-60822). To add the constraint, run the statement on the table.
- Since Spark 4.4, a write to a Data Source V2 table that parses the SQL text
of a CHECK constraint binds an unqualified built-in function name to
`system.builtin`, regardless of the SQL path and
`spark.sql.functionResolution.sessionOrder` (SPARK-60777). Previously, an
existing constraint could call another function with the same name.
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with `SUBQUERY_IN_CHECK_CONSTRAINT`
when the condition of a CHECK constraint contains a subquery. `ALTER TABLE ...
ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.HIGHER_ORDER_FUNCTION` on a subquery
in a lambda function. Previously, most of these statements failed with
`INTERNAL_ERROR`, but a subquery that needed no resolution, such as `id [...]
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
index d36bcdecfc12..7f6b1d34e23c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
@@ -119,6 +119,12 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
messageParameters = Map("checkCondition" -> check.condition)
)
}
+ check.child.collectFirst { case g: Generator => g }.foreach {
generator =>
+ generator.failAnalysis(
+ errorClass = "GENERATOR_IN_CHECK_CONSTRAINT",
+ messageParameters = Map("checkCondition" -> check.condition)
+ )
+ }
check.failIfCallsUserDefinedFunction()
case _ =>
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
index 9e44aa9b8bd0..78e94cac1c50 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
@@ -348,6 +348,25 @@ class CheckConstraintSuite extends QueryTest with
CommandSuiteBase with DDLComma
}
}
+ test("SPARK-60821: Check constraint cannot call a generator function") {
+ withNamespaceAndTable("ns", "tbl", nonPartitionCatalog) { t =>
+ sql(s"CREATE TABLE $t (id INT) $defaultUsing")
+ val call = "explode(array(id))"
+ val cond = s"$call IS NOT NULL"
+ Seq(
+ s"CREATE TABLE ${t}_2 (id INT, CONSTRAINT c1 CHECK ($cond))
$defaultUsing",
+ s"REPLACE TABLE $t (id INT, CONSTRAINT c1 CHECK ($cond)) $defaultUsing"
+ ).foreach { query =>
+ checkError(
+ exception = intercept[AnalysisException](sql(query)),
+ condition = "GENERATOR_IN_CHECK_CONSTRAINT",
+ sqlState = "42621",
+ parameters = Map("checkCondition" -> cond),
+ context = ExpectedContext(fragment = call))
+ }
+ }
+ }
+
test("SPARK-60777: Existing check constraint calling a user-defined function
still works") {
withUserDefinedFunction("tmp_len" -> true) {
spark.udf.register("tmp_len", (s: String) => s.length)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]