dongjoon-hyun opened a new pull request, #59992:
URL: https://github.com/apache/spark/pull/59992
### What changes were proposed in this pull request?
This PR makes the DDL reject a CHECK constraint whose condition calls a
user-defined function, and makes a write bind the built-in functions of a
condition it parses to `system.builtin`.
CHECK constraints exist only on DSv2 tables. V1 and Hive tables already fail
with `UNSUPPORTED_FEATURE.TABLE_OPERATION`.
1. `CREATE TABLE`, `REPLACE TABLE` and `ALTER TABLE ... ADD CONSTRAINT` fail
with the new error condition `USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT`. The
check covers:
- `UserDefinedExpression`: Scala, Java, Python and Hive functions, and V2
functions implementing `produceResult`.
- `Invoke` and `StaticInvoke`: V2 functions implementing an `invoke`
method or a static `invoke` method, which are not `UserDefinedExpression`.
- `V2Aggregator`: V2 aggregate functions.
- `SQLFunctionExpression`: SQL functions. `ALTER TABLE` already rejected
them with `UNSUPPORTED_SQL_UDF_USAGE`, and now reports the new error condition
like `CREATE TABLE` and `REPLACE TABLE`.
2. `ResolveTableConstraints` rewrites an unqualified `UnresolvedFunction` of
a parsed condition to `system.builtin.<name>` when `<name>` is a built-in
function.
3. The SQL migration guide describes these changes.
The check is in the DDL only, so an existing constraint that calls a
user-defined function keeps working. A built-in function called through an
`IDENTIFIER` clause, such as `IDENTIFIER('upper')(s)`, is not rewritten because
the parser does not produce an `UnresolvedFunction` for it.
### Why are the changes needed?
A CHECK condition is stored as SQL text. A write parses that text again
unless the stored V2 predicate converts back to a Catalyst expression, so
function names are resolved in the session of the write. The `Check` Javadoc
says that a condition cannot contain UDFs, but the DDL accepts them, and the
enforced rule can change silently:
```sql
-- a.ns.strlen and b.ns.strlen are different V2 functions with the same name
USE a.ns;
ALTER TABLE cat.ns.t ADD CONSTRAINT c CHECK (strlen(name) < 10);
INSERT INTO cat.ns.t SELECT 1, 'abcdefghijklmnop'; --
CHECK_CONSTRAINT_VIOLATION
USE b.ns;
INSERT INTO cat.ns.t SELECT 2, 'abcdefghijklmnop'; -- the violating row is
stored
```
- A temporary function of the writer shadows an unqualified V2 function in
the same way.
- Dropping the function makes every write fail with `UNRESOLVED_ROUTINE`,
and replacing it changes the rule silently.
- A SQL function accepted by `CREATE TABLE` makes every later write fail
with `Max iterations (100) reached for batch Resolution`.
Built-in functions are affected too. `upper(s) <> 'X'` has no V2 form that
converts back, so a write parses it. With `SET PATH = other_catalog.ns,
system.builtin` or `spark.sql.functionResolution.sessionOrder=first`, the name
binds to another function and the violating row is stored.
### Does this PR introduce _any_ user-facing change?
Yes.
- Before, `CREATE TABLE`, `REPLACE TABLE` and `ALTER TABLE ... ADD
CONSTRAINT` accepted a CHECK condition that calls a user-defined function,
except that `ALTER TABLE` rejected SQL functions. After, they fail:
```
[USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT] The check constraint
`tmp_len(name) < 10` calls the user-defined function `tmp_len`. Check
constraints can only call built-in functions. SQLSTATE: 42621
```
- `ALTER TABLE ... ADD CONSTRAINT` with a SQL function reports the error
above instead of `UNSUPPORTED_SQL_UDF_USAGE`.
- A write always binds a built-in function of a parsed condition to
`system.builtin`, regardless of its SQL path and
`spark.sql.functionResolution.sessionOrder`. Before, a violating row could be
stored under those settings.
Existing constraints that call a user-defined function are not rejected, and
their functions are still resolved in the session of the write. Released Spark
versions accept these conditions since Apache Spark 4.1.0.
### How was this patch tested?
Pass the CIs with the newly added test cases in `CheckConstraintSuite`:
- Rejection in `CREATE TABLE`, `REPLACE TABLE` and `ALTER TABLE` for each
kind of function.
- An existing constraint that calls a user-defined function stays enforced.
- A built-in function is not shadowed under `SET PATH` and
`spark.sql.functionResolution.sessionOrder=first`.
- Built-in functions such as `exists` and `extract` stay enforced after the
rewrite.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
