This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4fecd73323f7 [SPARK-60777][SQL] Reject user-defined functions in DSv2
CHECK constraints
4fecd73323f7 is described below
commit 4fecd73323f7598745276b336c1cbea9ee824257
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 15:38:41 2026 -0700
[SPARK-60777][SQL] Reject user-defined functions in DSv2 CHECK constraints
### What changes were proposed in this pull request?
This PR makes the DDL reject a CHECK constraint whose condition calls a
user-defined function, and makes a write bind the built-in functions of a
condition it parses to `system.builtin`.
CHECK constraints exist only on DSv2 tables. V1 and Hive tables already
fail with `UNSUPPORTED_FEATURE.TABLE_OPERATION`.
1. `CREATE TABLE`, `REPLACE TABLE` and `ALTER TABLE ... ADD CONSTRAINT`
fail with the new error condition `USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT`,
and the query context of the error points at the call. The check covers:
- `UserDefinedExpression`: Scala, Java, Python and Hive functions, and
V2 functions implementing `produceResult`.
- `Invoke` and `StaticInvoke`: V2 functions implementing an `invoke`
method or a static `invoke` method, which are not `UserDefinedExpression`.
- `V2Aggregator`: V2 aggregate functions in `CREATE TABLE` and `REPLACE
TABLE`. `ALTER TABLE` keeps rejecting them with `INVALID_WHERE_CONDITION`.
- `SQLFunctionExpression`: SQL functions in `CREATE TABLE` and `REPLACE
TABLE`. `ALTER TABLE` keeps rejecting them with `UNSUPPORTED_SQL_UDF_USAGE`.
2. `ResolveTableConstraints` rewrites an unqualified `UnresolvedFunction`
of a parsed condition to `system.builtin.<name>` when `<name>` is a built-in
function.
3. The SQL migration guide describes these changes.
The check is in the DDL only, so an existing constraint that calls a
user-defined function keeps working. The rewrite does not cover a built-in
function called through an `IDENTIFIER` clause, such as
`IDENTIFIER('upper')(s)`, because the parser does not produce an
`UnresolvedFunction` for it, or a function injected with
`SparkSessionExtensions.injectFunction`, which is not in
`FunctionRegistry.builtin`.
### Why are the changes needed?
A CHECK condition is stored as SQL text. A write parses that text again
unless the stored V2 predicate converts back to a Catalyst expression, so
function names are resolved in the session of the write. The `Check` Javadoc
says that a condition cannot contain UDFs, but the DDL accepts them, and the
enforced rule can change silently:
```sql
-- a.ns.strlen and b.ns.strlen are different V2 functions with the same name
USE a.ns;
ALTER TABLE cat.ns.t ADD CONSTRAINT c CHECK (strlen(name) < 10);
INSERT INTO cat.ns.t SELECT 1, 'abcdefghijklmnop'; --
CHECK_CONSTRAINT_VIOLATION
USE b.ns;
INSERT INTO cat.ns.t SELECT 2, 'abcdefghijklmnop'; -- the violating row is
stored
```
- A temporary function of the writer shadows an unqualified V2 function in
the same way.
- Dropping the function makes every write fail with `UNRESOLVED_ROUTINE`,
and replacing it changes the rule silently.
- A SQL function accepted by `CREATE TABLE` makes every later write fail
with `Max iterations (100) reached for batch Resolution`.
Built-in functions are affected too. `upper(s) <> 'X'` has no V2 form that
converts back, so a write parses it. With `SET PATH = other_catalog.ns,
system.builtin` or `spark.sql.functionResolution.sessionOrder=first`, the name
binds to another function and the violating row is stored.
### Does this PR introduce _any_ user-facing change?
Yes.
- Before, `CREATE TABLE` and `REPLACE TABLE` accepted a CHECK condition
that calls a user-defined function, and `ALTER TABLE ... ADD CONSTRAINT`
accepted it except for SQL and aggregate functions. After, the accepted
conditions fail:
```
[USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT] The check constraint
`tmp_len(name) < 10` calls the user-defined function `tmp_len`. Check
constraints can only call built-in functions. SQLSTATE: 42621
```
- A write always binds a built-in function of a parsed condition to
`system.builtin`, regardless of its SQL path and
`spark.sql.functionResolution.sessionOrder`. Before, a violating row could be
stored under those settings.
Existing constraints that call a user-defined function are not rejected,
and their functions are still resolved in the session of the write. Released
Spark versions accept these conditions since Apache Spark 4.1.0.
### How was this patch tested?
Pass the CIs with the newly added test cases in `CheckConstraintSuite`:
- Rejection in `CREATE TABLE` and `REPLACE TABLE` for each kind of
function, and in `ALTER TABLE` for the functions that it accepted before.
- An existing constraint that calls a user-defined function stays enforced.
- A built-in function is not shadowed under `SET PATH` and
`spark.sql.functionResolution.sessionOrder=first`.
- Built-in functions such as `exists` and `extract` stay enforced after the
rewrite.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59992 from dongjoon-hyun/SPARK-60777.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 6 +
docs/sql-migration-guide.md | 2 +
.../sql/catalyst/analysis/CheckAnalysis.scala | 3 +
.../analysis/ResolveTableConstraints.scala | 18 ++-
.../sql/catalyst/analysis/ResolveTableSpec.scala | 1 +
.../sql/catalyst/expressions/constraints.scala | 27 +++-
.../command/v2/CheckConstraintSuite.scala | 172 ++++++++++++++++++++-
7 files changed, 225 insertions(+), 4 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 5315dba509a9..1cd4360d4ee5 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -10196,6 +10196,12 @@
},
"sqlState" : "42601"
},
+ "USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT" : {
+ "message" : [
+ "The check constraint `<checkCondition>` calls the user-defined function
<functionName>. Check constraints can only call built-in functions."
+ ],
+ "sqlState" : "42621"
+ },
"USER_RAISED_EXCEPTION" : {
"message" : [
"<errorMessage>"
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 99798a3e5001..2d4bcec04262 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -67,6 +67,8 @@ license: |
- Since Spark 4.4, when a format is specified, `to_timestamp`,
`try_to_timestamp`, and `to_timestamp_ltz` cast a numeric first argument to a
string and parse it with the format, for example `to_timestamp(20201231,
'yyyyMMdd')` returns `2020-12-31 00:00:00`. This matches the existing behavior
when `spark.sql.timestampType` is `TIMESTAMP_NTZ`. Previously, with the default
`TIMESTAMP_LTZ` timestamp type, such calls failed with `INTERNAL_ERROR`. Note
that the string form of the number is par [...]
- Since Spark 4.4, the Spark Thrift Server logs a warning at startup when
`hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies
user identities (any recognized type other than `NONE`/`NOSASL`), because
executor-side data access still runs as the server's own service identity
rather than the connecting user (SPARK-5159). Set
`spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true` to acknowledge the
limitation and silence the warning. Spark 5.0 is expected to [...]
- Since Spark 4.4, `UPDATE` and `MERGE` on a Data Source V2 table fail with
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` when they assign to
a metadata column of the table. This includes `UPDATE SET *` and `INSERT *` in
`MERGE WITH SCHEMA EVOLUTION` when the source has a column with the name of a
metadata column of the target table. Previously, these assignments passed
analysis and were usually ignored. To restore the previous behavior, set
`spark.sql.legacy.ignoreMetadata [...]
+- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with
`USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT` when the condition of a CHECK
constraint calls a user-defined function, including a Data Source V2 function.
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with `UNSUPPORTED_SQL_UDF_USAGE`
on a SQL function and with `INVALID_WHERE_CONDITION` on an aggregate function.
Previously, a write could resolve the function name in [...]
+- Since Spark 4.4, a write to a Data Source V2 table that parses the SQL text
of a CHECK constraint binds an unqualified built-in function name to
`system.builtin`, regardless of the SQL path and
`spark.sql.functionResolution.sessionOrder` (SPARK-60777). Previously, an
existing constraint could call another function with the same name.
## Upgrading from Spark SQL 4.2 to 4.3
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index e92a811f041b..1dad9d634136 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -1052,6 +1052,9 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
messageParameters = Map("checkCondition" ->
a.checkConstraint.condition)
)
+ case a: AddCheckConstraint =>
+ a.checkConstraint.failIfCallsUserDefinedFunction()
+
case alter: AlterTableCommand =>
checkAlterTableCommand(alter)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
index 580af46533f3..303bc09125b9 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
@@ -18,6 +18,7 @@ package org.apache.spark.sql.catalyst.analysis
import scala.collection.mutable
+import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.expressions.{And, CheckInvariant,
EqualNullSafe, Expression, V2ExpressionUtils}
import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan,
V2WriteCommand, WriteDelta}
import org.apache.spark.sql.catalyst.rules.Rule
@@ -76,7 +77,22 @@ class ResolveTableConstraints(val catalogManager:
CatalogManager) extends Rule[L
private def buildCatalystExpression(c: Check): Expression = {
Option(c.predicate())
.flatMap(V2ExpressionUtils.toCatalyst)
-
.getOrElse(catalogManager.v1SessionCatalog.parser.parseExpression(c.predicateSql()))
+ .getOrElse(qualifyBuiltinFunctions(
+
catalogManager.v1SessionCatalog.parser.parseExpression(c.predicateSql())))
+ }
+
+ /**
+ * Qualifies the built-in functions a parsed check condition calls, so that
a temporary function
+ * or the SQL path of the session of the write cannot shadow them. Any other
name is left as is:
+ * a user-defined function in a constraint that the DDL no longer accepts
still resolves here.
+ * A function injected with `SparkSessionExtensions.injectFunction` is not in
+ * `FunctionRegistry.builtin`, so it is left as is too and can still be
shadowed.
+ */
+ private def qualifyBuiltinFunctions(condition: Expression): Expression =
condition.transform {
+ case f @ UnresolvedFunction(Seq(name), _, _, _, _, _, _, _)
+ if FunctionRegistry.builtin.functionExists(FunctionIdentifier(name)) =>
+ f.copy(nameParts =
+ Seq(CatalogManager.SYSTEM_CATALOG_NAME,
CatalogManager.BUILTIN_NAMESPACE, name))
}
/**
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
index 2b66ef503822..5371a87432ba 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
@@ -95,6 +95,7 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
messageParameters = Map("checkCondition" -> check.condition)
)
}
+ check.failIfCallsUserDefinedFunction()
case _ =>
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
index 12cfab81f959..fb910c6fc82c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
@@ -19,13 +19,17 @@ package org.apache.spark.sql.catalyst.expressions
import java.util.UUID
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.analysis.{TypeCheckResult,
UnresolvedException}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisErrorAt,
SQLFunctionExpression, TypeCheckResult, UnresolvedException}
+import org.apache.spark.sql.catalyst.expressions.aggregate.V2Aggregator
import org.apache.spark.sql.catalyst.expressions.codegen.{Block,
CodegenContext, ExprCode, FalseLiteral, TrueLiteral}
import org.apache.spark.sql.catalyst.expressions.codegen.Block.BlockHelper
+import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
import org.apache.spark.sql.catalyst.parser.ParseException
import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
import org.apache.spark.sql.catalyst.util.V2ExpressionBuilder
import org.apache.spark.sql.connector.catalog.constraints.Constraint
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
import org.apache.spark.sql.connector.expressions.FieldReference
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.types.{AbstractDataType, BooleanType, DataType}
@@ -128,6 +132,27 @@ case class CheckConstraint(
override def inputTypes: Seq[AbstractDataType] = Seq(BooleanType)
+ /**
+ * Fails the analysis if the condition calls a user-defined function. A
write can read `condition`
+ * again and resolve the name of the function in its own session, where the
name can refer to
+ * another function or to none, so the constraint would not enforce what
this DDL declares.
+ */
+ def failIfCallsUserDefinedFunction(): Unit = {
+ child.collectFirst {
+ case f: SQLFunctionExpression => (f, toSQLId(f.function.name.nameParts))
+ case f: UserDefinedExpression => (f, toSQLId(Seq(f.name)))
+ case i: StaticInvoke if i.scalarFunction.isDefined =>
+ (i, toSQLId(Seq(i.scalarFunction.get.name)))
+ case i @ Invoke(Literal(f: ScalarFunction[_], _), _, _, _, _, _, _, _) =>
+ (i, toSQLId(Seq(f.name)))
+ case a: V2Aggregator[_, _] => (a, toSQLId(Seq(a.aggrFunc.name)))
+ }.foreach { case (call, functionName) =>
+ call.failAnalysis(
+ errorClass = "USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT",
+ messageParameters = Map("checkCondition" -> condition, "functionName"
-> functionName))
+ }
+ }
+
def toV2Constraint: Constraint = {
val predicate = new V2ExpressionBuilder(child,
true).buildPredicate().orNull
val enforced = userProvidedCharacteristic.enforced.getOrElse(true)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
index 52da4802ef14..9a286a14b8dd 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CheckConstraintSuite.scala
@@ -17,15 +17,31 @@
package org.apache.spark.sql.execution.command.v2
+import test.org.apache.spark.sql.connector.catalog.functions.JavaAverage
+import
test.org.apache.spark.sql.connector.catalog.functions.JavaStrLen.JavaStrLenStaticMagic
+
import org.apache.spark.SparkRuntimeException
import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.plans.logical.Filter
import org.apache.spark.sql.catalyst.util.AttributeNameParser
import org.apache.spark.sql.catalyst.util.QuotingUtils.quoteNameParts
-import org.apache.spark.sql.connector.catalog.Table
-import org.apache.spark.sql.connector.catalog.constraints.Check
+import org.apache.spark.sql.connector.{StrLen, StrLenDefault, StrLenMagic}
+import org.apache.spark.sql.connector.catalog.{Identifier, InMemoryCatalog,
Table, TableCatalog, TableChange}
+import org.apache.spark.sql.connector.catalog.constraints.{Check, Constraint}
+import org.apache.spark.sql.connector.catalog.functions.{ScalarFunction,
UnboundFunction}
import org.apache.spark.sql.execution.command.DDLCommandTestUtils
import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{DataType, StringType}
+import org.apache.spark.unsafe.types.UTF8String
+
+// A V2 function that a test binds to the name of a built-in function to
shadow it.
+private object ConstantString extends ScalarFunction[UTF8String] {
+ override def inputTypes(): Array[DataType] = Array(StringType)
+ override def resultType(): DataType = StringType
+ override def name(): String = "constant_string"
+ override def produceResult(input: InternalRow): UTF8String =
UTF8String.fromString("OK")
+}
class CheckConstraintSuite extends QueryTest with CommandSuiteBase with
DDLCommandTestUtils {
override protected def command: String = "Check CONSTRAINT"
@@ -170,6 +186,158 @@ class CheckConstraintSuite extends QueryTest with
CommandSuiteBase with DDLComma
}
}
+ private def withV2Function(name: String, function: UnboundFunction)(f: =>
Unit): Unit = {
+ val funCatalog = spark.sessionState.catalogManager.catalog(s"fun_$catalog")
+ .asInstanceOf[InMemoryCatalog]
+ val ident = Identifier.of(Array("ns"), name)
+ withNamespace(s"fun_$catalog.ns") {
+ sql(s"CREATE NAMESPACE fun_$catalog.ns")
+ funCatalog.createFunction(ident, function)
+ try f finally funCatalog.dropFunction(ident)
+ }
+ }
+
+ // CREATE TABLE, REPLACE TABLE and, if `inAlter`, ALTER TABLE fail at the
call in the condition.
+ private def checkUserDefinedFunctionRejected(
+ cond: String,
+ call: String,
+ functionName: String,
+ inAlter: Boolean = true): Unit = {
+ withNamespaceAndTable("ns", "tbl", nonPartitionCatalog) { t =>
+ sql(s"CREATE TABLE $t (id INT, name STRING) $defaultUsing")
+ val alter = if (inAlter) Seq(s"ALTER TABLE $t ADD CONSTRAINT c1 CHECK
($cond)") else Nil
+ (alter ++ Seq(
+ s"CREATE TABLE ${t}_2 (id INT, name STRING, CONSTRAINT c1 CHECK
($cond)) $defaultUsing",
+ s"REPLACE TABLE $t (id INT, name STRING, CONSTRAINT c1 CHECK ($cond))
$defaultUsing"
+ )).foreach { query =>
+ checkError(
+ exception = intercept[AnalysisException](sql(query)),
+ condition = "USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT",
+ sqlState = "42621",
+ parameters = Map("checkCondition" -> cond, "functionName" ->
functionName),
+ context = ExpectedContext(fragment = call))
+ }
+ }
+ }
+
+ test("SPARK-60777: Check constraint cannot call a user-defined function") {
+ val strlen = s"fun_$catalog.ns.strlen"
+ Seq(
+ (StrLenDefault, s"$strlen(name) < 10", s"$strlen(name)",
"`strlen_default`"),
+ // The optimizer folds a call with constant arguments, but the DDL does
not.
+ (StrLenDefault, s"id < $strlen('abc')", s"$strlen('abc')",
"`strlen_default`"),
+ (StrLenMagic, s"$strlen(name) < 10", s"$strlen(name)", "`strlen_magic`"),
+ (new JavaStrLenStaticMagic, s"$strlen(name) < 10", s"$strlen(name)",
"`strlen`")
+ ).foreach { case (function, cond, call, functionName) =>
+ withV2Function("strlen", StrLen(function)) {
+ checkUserDefinedFunctionRejected(cond, call, functionName)
+ }
+ }
+
+ withUserDefinedFunction("tmp_len" -> true, "tmp_sql_len" -> true,
"sql_len" -> false) {
+ spark.udf.register("tmp_len", (s: String) => s.length)
+ checkUserDefinedFunctionRejected("tmp_len(name) < 10", "tmp_len(name)",
"`tmp_len`")
+ // ALTER TABLE keeps failing with UNSUPPORTED_SQL_UDF_USAGE on a SQL
function.
+ sql("CREATE TEMPORARY FUNCTION tmp_sql_len(x STRING) RETURNS INT RETURN
length(x)")
+ checkUserDefinedFunctionRejected(
+ "tmp_sql_len(name) < 10", "tmp_sql_len(name)", "`tmp_sql_len`",
inAlter = false)
+ sql("CREATE FUNCTION sql_len(x STRING) RETURNS INT RETURN length(x)")
+ checkUserDefinedFunctionRejected(
+ "sql_len(name) < 10",
+ "sql_len(name)",
+ "`spark_catalog`.`default`.`sql_len`",
+ inAlter = false)
+ }
+ }
+
+ test("SPARK-60777: Check constraint cannot call a V2 aggregate function") {
+ withV2Function("avg", new JavaAverage) {
+ // ALTER TABLE keeps failing with the error for an aggregate in a filter.
+ val call = s"fun_$catalog.ns.avg(CAST(id AS DOUBLE))"
+ checkUserDefinedFunctionRejected(s"$call > 0", call, "`davg`", inAlter =
false)
+ }
+ }
+
+ test("SPARK-60777: Existing check constraint calling a user-defined function
still works") {
+ withUserDefinedFunction("tmp_len" -> true) {
+ spark.udf.register("tmp_len", (s: String) => s.length)
+ withNamespaceAndTable("ns", "tbl", nonPartitionCatalog) { t =>
+ sql(s"CREATE TABLE $t (id INT, name STRING) $defaultUsing")
+ val check = Constraint.check("c1").predicateSql("tmp_len(name) <
10").build()
+
spark.sessionState.catalogManager.catalog(nonPartitionCatalog).asInstanceOf[TableCatalog]
+ .alterTable(Identifier.of(Array("ns"), "tbl"),
TableChange.addConstraint(check, null))
+ sql(s"INSERT INTO $t VALUES (1, 'abc')")
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ sql(s"INSERT INTO $t VALUES (2, 'abcdefghijklmnop')")
+ },
+ condition = "CHECK_CONSTRAINT_VIOLATION",
+ sqlState = "23001",
+ parameters = Map(
+ "constraintName" -> "c1",
+ "expression" -> "tmp_len(name) < 10",
+ "values" -> " - name : abcdefghijklmnop"))
+ checkAnswer(spark.table(t).select("id"), Row(1))
+ }
+ }
+ }
+
+ test("SPARK-60777: Check constraint is not shadowed in the session of a
write") {
+ def checkEnforced(t: String): Unit = {
+ // The settings of the write do shadow the built-in function outside of
the constraint.
+ checkAnswer(sql("SELECT upper('x')"), Row("OK"))
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ sql(s"INSERT INTO $t VALUES (1, 'x')")
+ },
+ condition = "CHECK_CONSTRAINT_VIOLATION",
+ sqlState = "23001",
+ parameters = Map(
+ "constraintName" -> "c1",
+ "expression" -> "UPPER(s) <> 'X'",
+ "values" -> " - s : x"))
+ checkAnswer(spark.table(t), Nil)
+ }
+
+ withV2Function("upper", StrLen(ConstantString)) {
+ withNamespaceAndTable("ns", "tbl", nonPartitionCatalog) { t =>
+ sql(s"CREATE TABLE $t (id INT, s STRING) $defaultUsing")
+ sql(s"ALTER TABLE $t ADD CONSTRAINT c1 CHECK (UPPER(s) <> 'X')")
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ sql(s"SET PATH = fun_$catalog.ns, system.builtin")
+ try checkEnforced(t) finally sql("SET PATH = DEFAULT_PATH")
+ }
+ spark.udf.register("upper", (_: String) => "OK")
+ try {
+ withSQLConf(SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key ->
"first") {
+ checkEnforced(t)
+ }
+ } finally {
+ sql("DROP TEMPORARY FUNCTION upper")
+ }
+ }
+ }
+ }
+
+ test("SPARK-60777: Check constraint calling built-in functions is enforced")
{
+ Seq(
+ "exists(a, x -> upper(CAST(x AS STRING)) = '2')",
+ "struct(id AS x).x > 0",
+ "extract(YEAR FROM d) < 2100"
+ ).foreach { cond =>
+ withNamespaceAndTable("ns", "tbl", nonPartitionCatalog) { t =>
+ sql(s"CREATE TABLE $t (id INT, a ARRAY<INT>, d DATE) $defaultUsing")
+ sql(s"ALTER TABLE $t ADD CONSTRAINT c1 CHECK ($cond)")
+ sql(s"INSERT INTO $t VALUES (1, array(2), DATE'2020-01-01')")
+ val error = intercept[SparkRuntimeException] {
+ sql(s"INSERT INTO $t VALUES (-1, array(-1), DATE'2999-01-01')")
+ }
+ assert(error.getCondition == "CHECK_CONSTRAINT_VIOLATION", cond)
+ checkAnswer(spark.table(t).select("id"), Row(1))
+ }
+ }
+ }
+
def getConstraintCharacteristics(): Seq[(String, String)] = {
Seq(
("", s"ENFORCED NORELY"),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]