This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 33ebee39ef34 [SPARK-60821][SQL] Fix DSv2 CREATE/REPLACE TABLE 
accepting a CHECK constraint that calls a generator function
33ebee39ef34 is described below

commit 33ebee39ef340ad74a58bb2d170ade0b877e8604
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 14:50:46 2026 -0800

    [SPARK-60821][SQL] Fix DSv2 CREATE/REPLACE TABLE accepting a CHECK 
constraint that calls a generator function
    
    ### What changes were proposed in this pull request?
    
    This PR makes `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2 table 
reject a CHECK constraint whose condition calls a generator function, such as 
`explode`, `posexplode`, and `inline`. They fail with the new error condition 
`GENERATOR_IN_CHECK_CONSTRAINT`, and the query context of the error points at 
the call.
    
    1. `ResolveTableSpec` looks for a `Generator` in the resolved condition. It 
does this after the check for aggregate and window functions and before 
`CheckConstraint.failIfCallsUserDefinedFunction`, so a Hive UDTF fails like a 
built-in generator function.
    2. The SQL migration guide describes this change.
    
    `ALTER TABLE ... ADD CONSTRAINT` already rejects these conditions, and this 
PR does not change its error. The check is in the DDL only, so a write to a 
table with an existing constraint fails as before.
    
    The error of `ALTER TABLE` is not reused because it shows the internal plan 
instead of the condition:
    
    ```
    [UNSUPPORTED_GENERATOR.OUTSIDE_SELECT] The generator is not supported: 
outside the SELECT clause, found: Filter NOT isnotnull(explode(array(id#12))). 
SQLSTATE: 42K0E
    ```
    
    `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` is not reused either, 
because an aggregate or window function reads multiple rows, while a generator 
function returns multiple rows.
    
    ### Why are the changes needed?
    
    `CREATE TABLE` and `REPLACE TABLE` accept a generator function in a CHECK 
constraint, but no write to the table can succeed after that:
    
    ```sql
    CREATE TABLE cat.ns.t (id INT, CONSTRAINT c1 CHECK (explode(array(id)) IS 
NOT NULL)) USING foo;  -- succeeds
    INSERT INTO cat.ns.t VALUES (1);  -- UNSUPPORTED_GENERATOR.OUTSIDE_SELECT
    ```
    
    https://github.com/apache/spark/pull/60005 rejected aggregate and window 
functions in these statements, but not generator functions.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes.
    
    - Before, `CREATE TABLE` and `REPLACE TABLE` accepted these conditions, and 
every later write to the table failed. After, the statements fail:
    
      ```
      [GENERATOR_IN_CHECK_CONSTRAINT] The check constraint `explode(array(id)) 
IS NOT NULL` calls a generator function. Check constraints compute one value 
for each row and cannot call functions that return multiple rows. SQLSTATE: 
42621
      ```
    
    - Compared to the unreleased master branch, a Hive UDTF in `CREATE TABLE` 
and `REPLACE TABLE` fails with this error instead of 
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT`.
    - `CREATE TABLE` on a V1 table with such a condition fails with this error 
instead of `UNSUPPORTED_FEATURE.TABLE_OPERATION`.
    
    Existing constraints that call a generator function are not rejected, and a 
write to their tables keeps failing with the same error. Released Spark 
versions accept these conditions since Apache Spark 4.1.0.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case in `CheckConstraintSuite`. It 
covers `CREATE TABLE` and `REPLACE TABLE` for `explode`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60039 from dongjoon-hyun/SPARK-60821.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit d4b33cad361dfa0c6c2f2ea95bac4fe8fe091a73)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../src/main/resources/error/error-conditions.json    |  6 ++++++
 docs/sql-migration-guide.md                           |  1 +
 .../sql/catalyst/analysis/ResolveTableSpec.scala      |  6 ++++++
 .../execution/command/v2/CheckConstraintSuite.scala   | 19 +++++++++++++++++++
 4 files changed, 32 insertions(+)

diff --git a/common/utils/src/main/resources/error/error-conditions.json 
b/common/utils/src/main/resources/error/error-conditions.json
index 0a490786a155..127be3eacb5a 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -3102,6 +3102,12 @@
     ],
     "sqlState" : "42623"
   },
+  "GENERATOR_IN_CHECK_CONSTRAINT" : {
+    "message" : [
+      "The check constraint `<checkCondition>` calls a generator function. 
Check constraints compute one value for each row and cannot call functions that 
return multiple rows."
+    ],
+    "sqlState" : "42621"
+  },
   "GEO_ENCODER_SRID_MISMATCH_ERROR" : {
     "message" : [
       "Failed to encode <type> value because provided SRID <valueSrid> of a 
value to encode does not match type SRID: <typeSrid>."
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 0c46f88db266..50a4ec4244be 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -69,6 +69,7 @@ license: |
 - Since Spark 4.4, `UPDATE` and `MERGE` on a Data Source V2 table fail with 
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` when they assign to 
a metadata column of the table. This includes `UPDATE SET *` and `INSERT *` in 
`MERGE WITH SCHEMA EVOLUTION` when the source has a column with the name of a 
metadata column of the target table. Previously, these assignments passed 
analysis and were usually ignored. To restore the previous behavior, set 
`spark.sql.legacy.ignoreMetadata [...]
 - Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD 
CONSTRAINT` on a Data Source V2 table fail with 
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK 
constraint calls a user-defined scalar function, including a Data Source V2 
function. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with 
`UNSUPPORTED_SQL_UDF_USAGE` on a SQL function. Previously, a write could 
resolve the function name in its own session, so that it enforced another rule 
or  [...]
 - Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2 
table fail with `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` when the 
condition of a CHECK constraint calls an aggregate or window function, 
including a user-defined aggregate function, `grouping`, and `grouping_id`. 
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with its existing errors, such 
as `INVALID_WHERE_CONDITION` and `WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE`. 
Previously, these statements succeeded and  [...]
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2 
table fail with `GENERATOR_IN_CHECK_CONSTRAINT` when the condition of a CHECK 
constraint calls a generator function, such as `explode`, `posexplode`, and 
`inline`. `ALTER TABLE ... ADD CONSTRAINT` keeps failing with 
`UNSUPPORTED_GENERATOR.OUTSIDE_SELECT`. Previously, these statements succeeded 
and every later write to the table failed (SPARK-60821). To run these 
statements, remove the function from the condition. E [...]
 - Since Spark 4.4, `ALTER TABLE ... ADD CONSTRAINT` with a CHECK constraint 
fails with `EXPECT_TABLE_NOT_VIEW.NO_ALTERNATIVE` when the target is a view, as 
it does with the other constraint types. Previously, when the view only 
projected or filtered a Data Source V2 table, the statement added the 
constraint to that table after validating only the rows of the view 
(SPARK-60822). To add the constraint, run the statement on the table.
 - Since Spark 4.4, a write to a Data Source V2 table that parses the SQL text 
of a CHECK constraint binds an unqualified built-in function name to 
`system.builtin`, regardless of the SQL path and 
`spark.sql.functionResolution.sessionOrder` (SPARK-60777). Previously, an 
existing constraint could call another function with the same name.
 - Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD 
CONSTRAINT` on a Data Source V2 table fail with `SUBQUERY_IN_CHECK_CONSTRAINT` 
when the condition of a CHECK constraint contains a subquery. `ALTER TABLE ... 
ADD CONSTRAINT` keeps failing with 
`UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.HIGHER_ORDER_FUNCTION` on a subquery 
in a lambda function. Previously, most of these statements failed with 
`INTERNAL_ERROR`, but a subquery that needed no resolution, such as `id  [...]
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
index d36bcdecfc12..7f6b1d34e23c 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
@@ -119,6 +119,12 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
               messageParameters = Map("checkCondition" -> check.condition)
             )
           }
+          check.child.collectFirst { case g: Generator => g }.foreach { 
generator =>
+            generator.failAnalysis(
+              errorClass = "GENERATOR_IN_CHECK_CONSTRAINT",
+              messageParameters = Map("checkCondition" -> check.condition)
+            )
+          }
           check.failIfCallsUserDefinedFunction()
         case _ =>
       }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
index 9e44aa9b8bd0..78e94cac1c50 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
@@ -348,6 +348,25 @@ class CheckConstraintSuite extends QueryTest with 
CommandSuiteBase with DDLComma
     }
   }
 
+  test("SPARK-60821: Check constraint cannot call a generator function") {
+    withNamespaceAndTable("ns", "tbl", nonPartitionCatalog) { t =>
+      sql(s"CREATE TABLE $t (id INT) $defaultUsing")
+      val call = "explode(array(id))"
+      val cond = s"$call IS NOT NULL"
+      Seq(
+        s"CREATE TABLE ${t}_2 (id INT, CONSTRAINT c1 CHECK ($cond)) 
$defaultUsing",
+        s"REPLACE TABLE $t (id INT, CONSTRAINT c1 CHECK ($cond)) $defaultUsing"
+      ).foreach { query =>
+        checkError(
+          exception = intercept[AnalysisException](sql(query)),
+          condition = "GENERATOR_IN_CHECK_CONSTRAINT",
+          sqlState = "42621",
+          parameters = Map("checkCondition" -> cond),
+          context = ExpectedContext(fragment = call))
+      }
+    }
+  }
+
   test("SPARK-60777: Existing check constraint calling a user-defined function 
still works") {
     withUserDefinedFunction("tmp_len" -> true) {
       spark.udf.register("tmp_len", (s: String) => s.length)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to