This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 71137da171eb [SPARK-60866][SQL] Fix wrong DSv2 generated column values
when a write shadows a built-in function
71137da171eb is described below
commit 71137da171eb190b4cbedd8e12064d46698076b6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 10:40:43 2026 -0800
[SPARK-60866][SQL] Fix wrong DSv2 generated column values when a write
shadows a built-in function
### What changes were proposed in this pull request?
This PR fixes a correctness issue: a write to a DSv2 table stores a wrong
value in an auto-filled generated column, and checks a provided value against a
wrong value, when a function in the SQL path of the write shadows a built-in
function of the generation expression. It also makes the DDL reject a
generation expression that calls a V2 function with a magic method, whose name
a write also resolves in its own session.
Both reuse what SPARK-60777 added for CHECK constraints. This PR moves that
code to a new object, `StoredExpressionFunctions`, so that CHECK constraints
and generated columns share it.
1. `StoredExpressionFunctions.qualifyBuiltinFunctions` rewrites an
unqualified `UnresolvedFunction` to `system.builtin.<name>` when `<name>` is a
built-in function. `StoredExpressionText` now applies it to every text that it
parses. Every read of the stored text of a generation expression goes through
it: when a write auto-fills a missing generated column, when it checks a
provided value, and when it reads the text under its own session.
2. `CheckAnalysis` rejects a generation expression with
`StoredExpressionFunctions.findUserDefinedFunctionCall`, which
`CheckConstraint.failIfCallsUserDefinedFunction` now calls too. The previous
check matched only `UserDefinedExpression`, which misses `Invoke` and
`StaticInvoke`: V2 functions implementing an `invoke` method or a static
`invoke` method. This PR also corrects the comment above the check, which said
that all V2 functions are `UserDefinedExpression`.
3. The SQL migration guide describes these changes.
The check is in the DDL only, so an existing generated column that calls a
V2 function keeps working. Like the rewrite for CHECK constraints, this one
does not cover a built-in function called through an `IDENTIFIER` clause, such
as `IDENTIFIER('upper')(s)`, or a function injected with
`SparkSessionExtensions.injectFunction`.
### Why are the changes needed?
This is a correctness issue. A generation expression is stored as SQL text,
and a write parses that text again. So function names are resolved in the
session of the write, not in the session of the DDL. A function in the SQL path
can shadow a built-in function, and the write silently stores a wrong value:
```sql
-- testcat.ns.upper is a V2 function that always returns 'OK', and
spark.sql.path.enabled is true.
CREATE TABLE testcat.t (s STRING, g STRING GENERATED ALWAYS AS (upper(s)))
USING foo;
SET PATH = testcat.ns, system.builtin;
-- Issue 1: It should store g = 'X', but it stores g = 'OK'.
INSERT INTO testcat.t (s) VALUES ('x');
-- Issue 2: It should succeed, but it fails with CHECK_CONSTRAINT_VIOLATION.
INSERT INTO testcat.t VALUES ('y', 'Y');
-- Issue 3: It should fail with CHECK_CONSTRAINT_VIOLATION, but it stores g
= 'OK'.
INSERT INTO testcat.t VALUES ('z', 'OK');
```
In Apache Spark
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27), a
temporary function does the same with
`spark.sql.functionResolution.sessionOrder=first`. SPARK-60757 reads the text
under fixed settings, which stops the temporary function, but not the SQL path.
Both still shadow a built-in function when a write reads the text under its own
session: for a text that the fixed settings cannot resolve, and with
`spark.sql.legacy.useCurrentConfigsForGeneratedCol [...]
A generation expression can call only built-in functions, and `CREATE
TABLE` already rejected a V2 function implementing `produceResult`. The same
function implementing `invoke` passed the DDL, and then a write resolved its
name in its own session.
### Does this PR introduce _any_ user-facing change?
Yes.
- A write always binds a built-in function of a parsed generation
expression to `system.builtin`, regardless of its SQL path and
`spark.sql.functionResolution.sessionOrder`. Before, the value that it filled
in or checked could come from another function with the same name. Apache Spark
4.3.0, which added the auto-fill, already stores such values. Rows that are
already stored keep their values.
- Before, `CREATE TABLE` and `REPLACE TABLE` accepted a generation
expression that calls a V2 scalar function with a magic `invoke` method. After,
they fail as they do for other user-defined functions:
```
[UNSUPPORTED_EXPRESSION_GENERATED_COLUMN] Cannot create generated column
g with generation expression testcat.ns.strlen(s) because failed to resolve
`strlen_magic` to a built-in function. SQLSTATE: 42621
```
Existing generated columns that call a V2 function are not rejected, and
their functions are still resolved in the session of the write.
### How was this patch tested?
Pass the CIs with the newly added test cases in `GeneratedColumnWriteSuite`:
- `CREATE TABLE` rejects a V2 function implementing an `invoke` method or a
static `invoke` method.
- `SET PATH` does not shadow a built-in function, both when a write
auto-fills a value and when it checks a provided value.
Both test cases fail without the fix.
```
build/sbt 'sql/testOnly *GeneratedColumnWriteSuite *CheckConstraintSuite
*DataSourceV2DataFrameSuite'
build/sbt 'sql/testOnly *DataSourceV2SQLSuiteV1Filter -- -z "generation
expression"'
build/sbt 'catalyst/testOnly *GeneratedColumnExpressionSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60081 from dongjoon-hyun/SPARK-60866.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit f95f8f2b93ae9f1b48efc5ef3b0c8ceaae0b85a5)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 2 +
.../sql/catalyst/analysis/CheckAnalysis.scala | 10 ++-
.../analysis/ResolveTableConstraints.scala | 17 +----
.../analysis/StoredExpressionFunctions.scala | 62 +++++++++++++++++
.../sql/catalyst/expressions/constraints.scala | 21 ++----
.../sql/catalyst/util/StoredExpressionText.scala | 6 +-
.../sql/connector/GeneratedColumnWriteSuite.scala | 79 +++++++++++++++++++++-
7 files changed, 156 insertions(+), 41 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index f853d80c42a5..dc7469c8d052 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -31,6 +31,8 @@ license: |
- Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic
Python UDF filter predicates that reference only partition columns. Previously,
these predicates could be dropped, returning rows that did not satisfy the
filter (SPARK-60486). Data derived from affected query results may need to be
recomputed.
- Since Spark 4.4, when a file source reads a partition column with a
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory
names, a value that cannot be cast to that type fails with
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously,
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such
as `2020-01-01 25:00:00`, became the start of that da [...]
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
+- Since Spark 4.4, a write to a Data Source V2 table that parses the SQL text
of a generation expression binds an unqualified built-in function name to
`system.builtin`, regardless of the SQL path and
`spark.sql.functionResolution.sessionOrder`. Previously, a temporary function
or a function in the SQL path with the same name could shadow the built-in
function, so that the write could silently store a value of an auto-filled
generated column that does not match the generation expression, [...]
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` when a generation
expression calls a Data Source V2 scalar function that has a magic `invoke`
method. Previously, they accepted such a call, and a write could resolve the
function name in its own session, so that it stored another value or failed
(SPARK-60866). To run these statements, change the expression to call only
built-in functions. Existing generated [...]
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a column referenced in a lambda function, such as `i` in `transform(a, x
-> x + i)`, or with `IDENTIFIER`, such as `IDENTIFIER('i')`, from the value
stored in that column. Previously, it read the value of the query before the
store assignment cast, so it could silently store a value that does not match
the stored column, and a later write of that row failed with
`CHECK_CONSTRAINT_VIOLATION` (SPARK-60817). [...]
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a field of another column, such as `s.f`, from the value stored in that
column. Previously, it read the value of the query before the store assignment
cast, so it could silently store a value that does not match the stored column,
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION`
(SPARK-60803). Rows that are already stored keep their values, and data derived
from them may need to be re [...]
- Since Spark 4.4, `spark.addArtifact` and `spark.addArtifacts` take a file
whose name ends with `.sparkpkg` as a [native data source
package](sql-native-data-sources.html), also when it is added with `file=True`,
`pyfile=True` or `archive=True` in Python without Spark Connect, or with
`file=True` from a Spark Connect client, and fail with
`INVALID_NATIVE_DATA_SOURCE_PACKAGE` for such a file that is not a valid
package, or whose version Spark does not support, or that has no library for
[...]
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index d25b15fdf091..5efe85190edb 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -1037,13 +1037,11 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
// not built-in already fail earlier in analysis: unknown
functions with
// UNRESOLVED_ROUTINE and persistent SQL functions with
UNSUPPORTED_SQL_UDF_USAGE.
// Here we only need to reject user-defined functions
(Scala/Python/Java/Hive UDFs
- // and v2 catalog functions, all of which are
`UserDefinedExpression`), which
- // resolve successfully.
- genExpr.child.foreach {
- case udf: UserDefinedExpression =>
+ // and v2 catalog functions), which resolve successfully.
+
StoredExpressionFunctions.findUserDefinedFunctionCall(genExpr.child).foreach {
+ case (_, functionName) =>
throw unsupportedExpressionError(
- s"failed to resolve `${udf.name}` to a built-in
function")
- case _ =>
+ s"failed to resolve $functionName to a built-in
function")
}
genExpr.validate(col.name, col.dataType, create.columns)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
index b961c5adfbcd..fdcd7017589c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
@@ -18,7 +18,6 @@ package org.apache.spark.sql.catalyst.analysis
import scala.collection.mutable
-import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.expressions.{And, Cast, CheckInvariant,
EqualNullSafe, Expression, V2ExpressionUtils}
import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan,
RowLevelWrite, V2WriteCommand, WriteDelta}
import org.apache.spark.sql.catalyst.rules.Rule
@@ -86,24 +85,10 @@ class ResolveTableConstraints(
private def buildCatalystExpression(c: Check): Expression = {
Option(c.predicate())
.flatMap(V2ExpressionUtils.toCatalyst)
- .getOrElse(qualifyBuiltinFunctions(
+ .getOrElse(StoredExpressionFunctions.qualifyBuiltinFunctions(
catalogManager.v1SessionCatalog.parser.parseExpression(c.predicateSql())))
}
- /**
- * Qualifies the built-in functions a parsed check condition calls, so that
a temporary function
- * or the SQL path of the session of the write cannot shadow them. Any other
name is left as is:
- * a user-defined function in a constraint that the DDL no longer accepts
still resolves here.
- * A function injected with `SparkSessionExtensions.injectFunction` is not in
- * `FunctionRegistry.builtin`, so it is left as is too and can still be
shadowed.
- */
- private def qualifyBuiltinFunctions(condition: Expression): Expression =
condition.transform {
- case f @ UnresolvedFunction(Seq(name), _, _, _, _, _, _, _)
- if FunctionRegistry.builtin.functionExists(FunctionIdentifier(name)) =>
- f.copy(nameParts =
- Seq(CatalogManager.SYSTEM_CATALOG_NAME,
CatalogManager.BUILTIN_NAMESPACE, name))
- }
-
/**
* For each generated column whose value was provided by the write, add a
CheckInvariant that
* validates the column value matches the generation expression. Generated
columns Spark
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StoredExpressionFunctions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StoredExpressionFunctions.scala
new file mode 100644
index 000000000000..29c02c1fe05c
--- /dev/null
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StoredExpressionFunctions.scala
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.catalyst.FunctionIdentifier
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal,
UserDefinedExpression}
+import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
+import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+
+/**
+ * The function calls of an expression that a table stores as SQL text, such
as the condition of a
+ * CHECK constraint or a generation expression. A write reads the text again
and resolves the names
+ * of the functions in its own session, so these calls must not depend on that
session.
+ */
+object StoredExpressionFunctions {
+
+ /**
+ * Returns the first call of a user-defined function in `expr` and the
quoted name of the
+ * function. In the session of a write, the name can refer to another
function or to none.
+ */
+ def findUserDefinedFunctionCall(expr: Expression): Option[(Expression,
String)] = {
+ expr.collectFirst {
+ case f: SQLFunctionExpression => (f, toSQLId(f.function.name.nameParts))
+ case f: UserDefinedExpression => (f, toSQLId(Seq(f.name)))
+ case i: StaticInvoke if i.scalarFunction.isDefined =>
+ (i, toSQLId(Seq(i.scalarFunction.get.name)))
+ case i @ Invoke(Literal(f: ScalarFunction[_], _), _, _, _, _, _, _, _) =>
+ (i, toSQLId(Seq(f.name)))
+ }
+ }
+
+ /**
+ * Qualifies the built-in functions that a parsed text calls, so that a
temporary function or the
+ * SQL path of the session of a write cannot shadow them. Any other name is
left as is: a
+ * user-defined function in a text that the DDL no longer accepts still
resolves here.
+ * A function injected with `SparkSessionExtensions.injectFunction` is not in
+ * `FunctionRegistry.builtin`, so it is left as is too and can still be
shadowed.
+ */
+ def qualifyBuiltinFunctions(expr: Expression): Expression = expr.transform {
+ case f @ UnresolvedFunction(Seq(name), _, _, _, _, _, _, _)
+ if FunctionRegistry.builtin.functionExists(FunctionIdentifier(name)) =>
+ f.copy(nameParts =
+ Seq(CatalogManager.SYSTEM_CATALOG_NAME,
CatalogManager.BUILTIN_NAMESPACE, name))
+ }
+}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
index 306ddc9ad825..8fdf886f1444 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
@@ -19,16 +19,13 @@ package org.apache.spark.sql.catalyst.expressions
import java.util.UUID
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.analysis.{AnalysisErrorAt,
SQLFunctionExpression, TypeCheckResult, UnresolvedException}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisErrorAt,
StoredExpressionFunctions, TypeCheckResult, UnresolvedException}
import org.apache.spark.sql.catalyst.expressions.codegen.{Block,
CodegenContext, ExprCode, FalseLiteral, TrueLiteral}
import org.apache.spark.sql.catalyst.expressions.codegen.Block.BlockHelper
-import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
import org.apache.spark.sql.catalyst.parser.ParseException
import org.apache.spark.sql.catalyst.trees.CurrentOrigin
-import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
import org.apache.spark.sql.catalyst.util.V2ExpressionBuilder
import org.apache.spark.sql.connector.catalog.constraints.Constraint
-import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
import org.apache.spark.sql.connector.expressions.FieldReference
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.types.{AbstractDataType, BooleanType, DataType}
@@ -137,17 +134,11 @@ case class CheckConstraint(
* another function or to none, so the constraint would not enforce what
this DDL declares.
*/
def failIfCallsUserDefinedFunction(): Unit = {
- child.collectFirst {
- case f: SQLFunctionExpression => (f, toSQLId(f.function.name.nameParts))
- case f: UserDefinedExpression => (f, toSQLId(Seq(f.name)))
- case i: StaticInvoke if i.scalarFunction.isDefined =>
- (i, toSQLId(Seq(i.scalarFunction.get.name)))
- case i @ Invoke(Literal(f: ScalarFunction[_], _), _, _, _, _, _, _, _) =>
- (i, toSQLId(Seq(f.name)))
- }.foreach { case (call, functionName) =>
- call.failAnalysis(
- errorClass = "USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT",
- messageParameters = Map("checkCondition" -> condition, "functionName"
-> functionName))
+ StoredExpressionFunctions.findUserDefinedFunctionCall(child).foreach {
+ case (call, functionName) =>
+ call.failAnalysis(
+ errorClass = "USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT",
+ messageParameters = Map("checkCondition" -> condition,
"functionName" -> functionName))
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
index cdf189812a40..7fd4c276b669 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.catalyst.util
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.analysis.{Analyzer,
ExpressionWithUnresolvedIdentifier}
+import org.apache.spark.sql.catalyst.analysis.{Analyzer,
ExpressionWithUnresolvedIdentifier, StoredExpressionFunctions}
import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute,
Expression, ExprId, NamedExpression}
import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation,
LogicalPlan, Project}
@@ -68,13 +68,15 @@ object StoredExpressionText {
def parseWithSessionAnsi(text: String): Expression =
parse(confWithSessionAnsi(SQLConf.get), text)
private def parse(c: SQLConf, text: String): Expression = {
- try {
+ val parsed = try {
SQLConf.withExistingConf(c) {
CatalystSqlParser.parseExpression(text)
}
} catch {
case _: ParseException => CatalystSqlParser.parseExpression(text)
}
+ // Neither the SQL path nor a temporary function of the session can shadow
a built-in function.
+ StoredExpressionFunctions.qualifyBuiltinFunctions(parsed)
}
private def confWithSessionAnsi(session: SQLConf): SQLConf = {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
index 6161a1562409..918b7f3e6eee 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
@@ -19,17 +19,29 @@ package org.apache.spark.sql.connector
import java.util.Collections
+import
test.org.apache.spark.sql.connector.catalog.functions.JavaStrLen.JavaStrLenStaticMagic
+
import org.apache.spark.{SparkRuntimeException, SparkThrowable}
import org.apache.spark.sql.{AnalysisException, DataFrame, QueryTest, Row}
-import org.apache.spark.sql.catalyst.QueryPlanningTracker
+import org.apache.spark.sql.catalyst.{InternalRow, QueryPlanningTracker}
import org.apache.spark.sql.catalyst.expressions.CheckInvariant
import org.apache.spark.sql.catalyst.util.GeneratedColumn
import org.apache.spark.sql.connector.catalog.{Column, GenerationExpression,
Identifier, InMemoryCatalog, InMemoryRowLevelOperationTableCatalog}
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.CatalogHelper
+import org.apache.spark.sql.connector.catalog.functions.{ScalarFunction,
UnboundFunction}
import org.apache.spark.sql.connector.expressions.{Expression => V2Expression,
GeneralScalarExpression, Transform}
import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{IntegerType, MetadataBuilder, StringType,
StructType}
+import org.apache.spark.sql.types.{DataType, IntegerType, MetadataBuilder,
StringType, StructType}
+import org.apache.spark.unsafe.types.UTF8String
+
+// A V2 function that a test binds to the name of a built-in function to
shadow it.
+private object ConstantString extends ScalarFunction[UTF8String] {
+ override def inputTypes(): Array[DataType] = Array(StringType)
+ override def resultType(): DataType = StringType
+ override def name(): String = "constant_string"
+ override def produceResult(input: InternalRow): UTF8String =
UTF8String.fromString("OK")
+}
/**
* Tests for generated column auto-fill and constraint enforcement during
writes.
@@ -459,6 +471,69 @@ class GeneratedColumnWriteSuite extends QueryTest with
DatasourceV2SQLBase {
}
}
+ private def addFunction(name: String, function: UnboundFunction): Unit = {
+ catalog("testcat").asInstanceOf[InMemoryCatalog]
+ .createFunction(Identifier.of(Array("ns"), name), function)
+ }
+
+ test("SPARK-60866: generation expression cannot call a V2 function with a
magic method") {
+ val tblName = "my_tab"
+ Seq(
+ StrLenMagic -> "`strlen_magic`",
+ new JavaStrLenStaticMagic -> "`strlen`"
+ ).foreach { case (function, functionName) =>
+ addFunction("strlen", StrLen(function))
+ withTable(s"testcat.$tblName") {
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | s STRING,
+ | g INT GENERATED ALWAYS AS (testcat.ns.strlen(s))
+ |) USING foo""".stripMargin)
+ },
+ condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ parameters = Map(
+ "fieldName" -> "g",
+ "expressionStr" -> "testcat.ns.strlen(s)",
+ "reason" -> s"failed to resolve $functionName to a built-in
function"))
+ }
+ }
+ }
+
+ test("SPARK-60866: SQL path does not shadow a built-in function of a
generation expression") {
+ val tblName = "my_tab"
+ addFunction("upper", StrLen(ConstantString))
+ withTable(s"testcat.$tblName") {
+ sql(s"""CREATE TABLE testcat.$tblName(
+ | s STRING,
+ | g STRING GENERATED ALWAYS AS (upper(s))
+ |) USING foo""".stripMargin)
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ sql("SET PATH = testcat.ns, system.builtin")
+ try {
+ // The path does shadow the built-in function outside of the
generation expression.
+ checkAnswer(sql("SELECT upper('x')"), Row("OK"))
+ sql(s"INSERT INTO testcat.$tblName(s) VALUES ('x')")
+ checkAnswer(spark.table(s"testcat.$tblName"), Row("x", "X"))
+ sql(s"INSERT INTO testcat.$tblName VALUES ('y', 'Y')")
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ sql(s"INSERT INTO testcat.$tblName VALUES ('z', 'OK')")
+ },
+ condition = "CHECK_CONSTRAINT_VIOLATION",
+ sqlState = "23001",
+ parameters = Map(
+ "constraintName" -> "Generated Column",
+ "expression" -> "g <=> upper(s)",
+ "values" -> " - g : OK"))
+ checkAnswer(spark.table(s"testcat.$tblName"), Seq(Row("x", "X"),
Row("y", "Y")))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ }
+ }
+ }
+ }
+
test("INSERT by position does not auto-fill trailing generated column") {
val tblName = "my_tab"
withTable(s"testcat.$tblName") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]