This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 71137da171eb [SPARK-60866][SQL] Fix wrong DSv2 generated column values 
when a write shadows a built-in function
71137da171eb is described below

commit 71137da171eb190b4cbedd8e12064d46698076b6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 10:40:43 2026 -0800

    [SPARK-60866][SQL] Fix wrong DSv2 generated column values when a write 
shadows a built-in function
    
    ### What changes were proposed in this pull request?
    
    This PR fixes a correctness issue: a write to a DSv2 table stores a wrong 
value in an auto-filled generated column, and checks a provided value against a 
wrong value, when a function in the SQL path of the write shadows a built-in 
function of the generation expression. It also makes the DDL reject a 
generation expression that calls a V2 function with a magic method, whose name 
a write also resolves in its own session.
    
    Both reuse what SPARK-60777 added for CHECK constraints. This PR moves that 
code to a new object, `StoredExpressionFunctions`, so that CHECK constraints 
and generated columns share it.
    
    1. `StoredExpressionFunctions.qualifyBuiltinFunctions` rewrites an 
unqualified `UnresolvedFunction` to `system.builtin.<name>` when `<name>` is a 
built-in function. `StoredExpressionText` now applies it to every text that it 
parses. Every read of the stored text of a generation expression goes through 
it: when a write auto-fills a missing generated column, when it checks a 
provided value, and when it reads the text under its own session.
    2. `CheckAnalysis` rejects a generation expression with 
`StoredExpressionFunctions.findUserDefinedFunctionCall`, which 
`CheckConstraint.failIfCallsUserDefinedFunction` now calls too. The previous 
check matched only `UserDefinedExpression`, which misses `Invoke` and 
`StaticInvoke`: V2 functions implementing an `invoke` method or a static 
`invoke` method. This PR also corrects the comment above the check, which said 
that all V2 functions are `UserDefinedExpression`.
    3. The SQL migration guide describes these changes.
    
    The check is in the DDL only, so an existing generated column that calls a 
V2 function keeps working. Like the rewrite for CHECK constraints, this one 
does not cover a built-in function called through an `IDENTIFIER` clause, such 
as `IDENTIFIER('upper')(s)`, or a function injected with 
`SparkSessionExtensions.injectFunction`.
    
    ### Why are the changes needed?
    
    This is a correctness issue. A generation expression is stored as SQL text, 
and a write parses that text again. So function names are resolved in the 
session of the write, not in the session of the DDL. A function in the SQL path 
can shadow a built-in function, and the write silently stores a wrong value:
    
    ```sql
    -- testcat.ns.upper is a V2 function that always returns 'OK', and 
spark.sql.path.enabled is true.
    CREATE TABLE testcat.t (s STRING, g STRING GENERATED ALWAYS AS (upper(s))) 
USING foo;
    SET PATH = testcat.ns, system.builtin;
    -- Issue 1: It should store g = 'X', but it stores g = 'OK'.
    INSERT INTO testcat.t (s) VALUES ('x');
    -- Issue 2: It should succeed, but it fails with CHECK_CONSTRAINT_VIOLATION.
    INSERT INTO testcat.t VALUES ('y', 'Y');
    -- Issue 3: It should fail with CHECK_CONSTRAINT_VIOLATION, but it stores g 
= 'OK'.
    INSERT INTO testcat.t VALUES ('z', 'OK');
    ```
    
    In Apache Spark 
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27), a 
temporary function does the same with 
`spark.sql.functionResolution.sessionOrder=first`. SPARK-60757 reads the text 
under fixed settings, which stops the temporary function, but not the SQL path. 
Both still shadow a built-in function when a write reads the text under its own 
session: for a text that the fixed settings cannot resolve, and with 
`spark.sql.legacy.useCurrentConfigsForGeneratedCol [...]
    
    A generation expression can call only built-in functions, and `CREATE 
TABLE` already rejected a V2 function implementing `produceResult`. The same 
function implementing `invoke` passed the DDL, and then a write resolved its 
name in its own session.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes.
    
    - A write always binds a built-in function of a parsed generation 
expression to `system.builtin`, regardless of its SQL path and 
`spark.sql.functionResolution.sessionOrder`. Before, the value that it filled 
in or checked could come from another function with the same name. Apache Spark 
4.3.0, which added the auto-fill, already stores such values. Rows that are 
already stored keep their values.
    - Before, `CREATE TABLE` and `REPLACE TABLE` accepted a generation 
expression that calls a V2 scalar function with a magic `invoke` method. After, 
they fail as they do for other user-defined functions:
    
      ```
      [UNSUPPORTED_EXPRESSION_GENERATED_COLUMN] Cannot create generated column 
g with generation expression testcat.ns.strlen(s) because failed to resolve 
`strlen_magic` to a built-in function. SQLSTATE: 42621
      ```
    
    Existing generated columns that call a V2 function are not rejected, and 
their functions are still resolved in the session of the write.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases in `GeneratedColumnWriteSuite`:
    
    - `CREATE TABLE` rejects a V2 function implementing an `invoke` method or a 
static `invoke` method.
    - `SET PATH` does not shadow a built-in function, both when a write 
auto-fills a value and when it checks a provided value.
    
    Both test cases fail without the fix.
    
    ```
    build/sbt 'sql/testOnly *GeneratedColumnWriteSuite *CheckConstraintSuite 
*DataSourceV2DataFrameSuite'
    build/sbt 'sql/testOnly *DataSourceV2SQLSuiteV1Filter -- -z "generation 
expression"'
    build/sbt 'catalyst/testOnly *GeneratedColumnExpressionSuite'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60081 from dongjoon-hyun/SPARK-60866.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit f95f8f2b93ae9f1b48efc5ef3b0c8ceaae0b85a5)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-migration-guide.md                        |  2 +
 .../sql/catalyst/analysis/CheckAnalysis.scala      | 10 ++-
 .../analysis/ResolveTableConstraints.scala         | 17 +----
 .../analysis/StoredExpressionFunctions.scala       | 62 +++++++++++++++++
 .../sql/catalyst/expressions/constraints.scala     | 21 ++----
 .../sql/catalyst/util/StoredExpressionText.scala   |  6 +-
 .../sql/connector/GeneratedColumnWriteSuite.scala  | 79 +++++++++++++++++++++-
 7 files changed, 156 insertions(+), 41 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index f853d80c42a5..dc7469c8d052 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -31,6 +31,8 @@ license: |
 - Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic 
Python UDF filter predicates that reference only partition columns. Previously, 
these predicates could be dropped, returning rows that did not satisfy the 
filter (SPARK-60486). Data derived from affected query results may need to be 
recomputed.
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory 
names, a value that cannot be cast to that type fails with 
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously, 
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it 
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such 
as `2020-01-01 25:00:00`, became the start of that da [...]
 - Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`, 
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of 
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so 
existing paths can match different keys or become invalid. For example, use 
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes 
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set 
`spark.sql.legacy.variantPathB [...]
+- Since Spark 4.4, a write to a Data Source V2 table that parses the SQL text 
of a generation expression binds an unqualified built-in function name to 
`system.builtin`, regardless of the SQL path and 
`spark.sql.functionResolution.sessionOrder`. Previously, a temporary function 
or a function in the SQL path with the same name could shadow the built-in 
function, so that the write could silently store a value of an auto-filled 
generated column that does not match the generation expression, [...]
+- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2 
table fail with `UNSUPPORTED_EXPRESSION_GENERATED_COLUMN` when a generation 
expression calls a Data Source V2 scalar function that has a magic `invoke` 
method. Previously, they accepted such a call, and a write could resolve the 
function name in its own session, so that it stored another value or failed 
(SPARK-60866). To run these statements, change the expression to call only 
built-in functions. Existing generated [...]
 - Since Spark 4.4, an auto-filled generated column of a Data Source V2 table 
reads a column referenced in a lambda function, such as `i` in `transform(a, x 
-> x + i)`, or with `IDENTIFIER`, such as `IDENTIFIER('i')`, from the value 
stored in that column. Previously, it read the value of the query before the 
store assignment cast, so it could silently store a value that does not match 
the stored column, and a later write of that row failed with 
`CHECK_CONSTRAINT_VIOLATION` (SPARK-60817).  [...]
 - Since Spark 4.4, an auto-filled generated column of a Data Source V2 table 
reads a field of another column, such as `s.f`, from the value stored in that 
column. Previously, it read the value of the query before the store assignment 
cast, so it could silently store a value that does not match the stored column, 
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION` 
(SPARK-60803). Rows that are already stored keep their values, and data derived 
from them may need to be re [...]
 - Since Spark 4.4, `spark.addArtifact` and `spark.addArtifacts` take a file 
whose name ends with `.sparkpkg` as a [native data source 
package](sql-native-data-sources.html), also when it is added with `file=True`, 
`pyfile=True` or `archive=True` in Python without Spark Connect, or with 
`file=True` from a Spark Connect client, and fail with 
`INVALID_NATIVE_DATA_SOURCE_PACKAGE` for such a file that is not a valid 
package, or whose version Spark does not support, or that has no library for  
[...]
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index d25b15fdf091..5efe85190edb 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -1037,13 +1037,11 @@ trait CheckAnalysis extends LookupCatalog with 
QueryErrorsBase with PlanToString
                 // not built-in already fail earlier in analysis: unknown 
functions with
                 // UNRESOLVED_ROUTINE and persistent SQL functions with 
UNSUPPORTED_SQL_UDF_USAGE.
                 // Here we only need to reject user-defined functions 
(Scala/Python/Java/Hive UDFs
-                // and v2 catalog functions, all of which are 
`UserDefinedExpression`), which
-                // resolve successfully.
-                genExpr.child.foreach {
-                  case udf: UserDefinedExpression =>
+                // and v2 catalog functions), which resolve successfully.
+                
StoredExpressionFunctions.findUserDefinedFunctionCall(genExpr.child).foreach {
+                  case (_, functionName) =>
                     throw unsupportedExpressionError(
-                      s"failed to resolve `${udf.name}` to a built-in 
function")
-                  case _ =>
+                      s"failed to resolve $functionName to a built-in 
function")
                 }
                 genExpr.validate(col.name, col.dataType, create.columns)
               }
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
index b961c5adfbcd..fdcd7017589c 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableConstraints.scala
@@ -18,7 +18,6 @@ package org.apache.spark.sql.catalyst.analysis
 
 import scala.collection.mutable
 
-import org.apache.spark.sql.catalyst.FunctionIdentifier
 import org.apache.spark.sql.catalyst.expressions.{And, Cast, CheckInvariant, 
EqualNullSafe, Expression, V2ExpressionUtils}
 import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, 
RowLevelWrite, V2WriteCommand, WriteDelta}
 import org.apache.spark.sql.catalyst.rules.Rule
@@ -86,24 +85,10 @@ class ResolveTableConstraints(
   private def buildCatalystExpression(c: Check): Expression = {
     Option(c.predicate())
       .flatMap(V2ExpressionUtils.toCatalyst)
-      .getOrElse(qualifyBuiltinFunctions(
+      .getOrElse(StoredExpressionFunctions.qualifyBuiltinFunctions(
         
catalogManager.v1SessionCatalog.parser.parseExpression(c.predicateSql())))
   }
 
-  /**
-   * Qualifies the built-in functions a parsed check condition calls, so that 
a temporary function
-   * or the SQL path of the session of the write cannot shadow them. Any other 
name is left as is:
-   * a user-defined function in a constraint that the DDL no longer accepts 
still resolves here.
-   * A function injected with `SparkSessionExtensions.injectFunction` is not in
-   * `FunctionRegistry.builtin`, so it is left as is too and can still be 
shadowed.
-   */
-  private def qualifyBuiltinFunctions(condition: Expression): Expression = 
condition.transform {
-    case f @ UnresolvedFunction(Seq(name), _, _, _, _, _, _, _)
-        if FunctionRegistry.builtin.functionExists(FunctionIdentifier(name)) =>
-      f.copy(nameParts =
-        Seq(CatalogManager.SYSTEM_CATALOG_NAME, 
CatalogManager.BUILTIN_NAMESPACE, name))
-  }
-
   /**
    * For each generated column whose value was provided by the write, add a 
CheckInvariant that
    * validates the column value matches the generation expression. Generated 
columns Spark
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StoredExpressionFunctions.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StoredExpressionFunctions.scala
new file mode 100644
index 000000000000..29c02c1fe05c
--- /dev/null
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StoredExpressionFunctions.scala
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.catalyst.FunctionIdentifier
+import org.apache.spark.sql.catalyst.expressions.{Expression, Literal, 
UserDefinedExpression}
+import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
+import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
+import org.apache.spark.sql.connector.catalog.CatalogManager
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
+
+/**
+ * The function calls of an expression that a table stores as SQL text, such 
as the condition of a
+ * CHECK constraint or a generation expression. A write reads the text again 
and resolves the names
+ * of the functions in its own session, so these calls must not depend on that 
session.
+ */
+object StoredExpressionFunctions {
+
+  /**
+   * Returns the first call of a user-defined function in `expr` and the 
quoted name of the
+   * function. In the session of a write, the name can refer to another 
function or to none.
+   */
+  def findUserDefinedFunctionCall(expr: Expression): Option[(Expression, 
String)] = {
+    expr.collectFirst {
+      case f: SQLFunctionExpression => (f, toSQLId(f.function.name.nameParts))
+      case f: UserDefinedExpression => (f, toSQLId(Seq(f.name)))
+      case i: StaticInvoke if i.scalarFunction.isDefined =>
+        (i, toSQLId(Seq(i.scalarFunction.get.name)))
+      case i @ Invoke(Literal(f: ScalarFunction[_], _), _, _, _, _, _, _, _) =>
+        (i, toSQLId(Seq(f.name)))
+    }
+  }
+
+  /**
+   * Qualifies the built-in functions that a parsed text calls, so that a 
temporary function or the
+   * SQL path of the session of a write cannot shadow them. Any other name is 
left as is: a
+   * user-defined function in a text that the DDL no longer accepts still 
resolves here.
+   * A function injected with `SparkSessionExtensions.injectFunction` is not in
+   * `FunctionRegistry.builtin`, so it is left as is too and can still be 
shadowed.
+   */
+  def qualifyBuiltinFunctions(expr: Expression): Expression = expr.transform {
+    case f @ UnresolvedFunction(Seq(name), _, _, _, _, _, _, _)
+        if FunctionRegistry.builtin.functionExists(FunctionIdentifier(name)) =>
+      f.copy(nameParts =
+        Seq(CatalogManager.SYSTEM_CATALOG_NAME, 
CatalogManager.BUILTIN_NAMESPACE, name))
+  }
+}
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
index 306ddc9ad825..8fdf886f1444 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala
@@ -19,16 +19,13 @@ package org.apache.spark.sql.catalyst.expressions
 import java.util.UUID
 
 import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.analysis.{AnalysisErrorAt, 
SQLFunctionExpression, TypeCheckResult, UnresolvedException}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisErrorAt, 
StoredExpressionFunctions, TypeCheckResult, UnresolvedException}
 import org.apache.spark.sql.catalyst.expressions.codegen.{Block, 
CodegenContext, ExprCode, FalseLiteral, TrueLiteral}
 import org.apache.spark.sql.catalyst.expressions.codegen.Block.BlockHelper
-import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.CurrentOrigin
-import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
 import org.apache.spark.sql.catalyst.util.V2ExpressionBuilder
 import org.apache.spark.sql.connector.catalog.constraints.Constraint
-import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
 import org.apache.spark.sql.connector.expressions.FieldReference
 import org.apache.spark.sql.errors.QueryExecutionErrors
 import org.apache.spark.sql.types.{AbstractDataType, BooleanType, DataType}
@@ -137,17 +134,11 @@ case class CheckConstraint(
    * another function or to none, so the constraint would not enforce what 
this DDL declares.
    */
   def failIfCallsUserDefinedFunction(): Unit = {
-    child.collectFirst {
-      case f: SQLFunctionExpression => (f, toSQLId(f.function.name.nameParts))
-      case f: UserDefinedExpression => (f, toSQLId(Seq(f.name)))
-      case i: StaticInvoke if i.scalarFunction.isDefined =>
-        (i, toSQLId(Seq(i.scalarFunction.get.name)))
-      case i @ Invoke(Literal(f: ScalarFunction[_], _), _, _, _, _, _, _, _) =>
-        (i, toSQLId(Seq(f.name)))
-    }.foreach { case (call, functionName) =>
-      call.failAnalysis(
-        errorClass = "USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT",
-        messageParameters = Map("checkCondition" -> condition, "functionName" 
-> functionName))
+    StoredExpressionFunctions.findUserDefinedFunctionCall(child).foreach {
+      case (call, functionName) =>
+        call.failAnalysis(
+          errorClass = "USER_DEFINED_FUNCTION_IN_CHECK_CONSTRAINT",
+          messageParameters = Map("checkCondition" -> condition, 
"functionName" -> functionName))
     }
   }
 
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
index cdf189812a40..7fd4c276b669 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StoredExpressionText.scala
@@ -18,7 +18,7 @@
 package org.apache.spark.sql.catalyst.util
 
 import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.analysis.{Analyzer, 
ExpressionWithUnresolvedIdentifier}
+import org.apache.spark.sql.catalyst.analysis.{Analyzer, 
ExpressionWithUnresolvedIdentifier, StoredExpressionFunctions}
 import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, 
Expression, ExprId, NamedExpression}
 import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
 import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, 
LogicalPlan, Project}
@@ -68,13 +68,15 @@ object StoredExpressionText {
   def parseWithSessionAnsi(text: String): Expression = 
parse(confWithSessionAnsi(SQLConf.get), text)
 
   private def parse(c: SQLConf, text: String): Expression = {
-    try {
+    val parsed = try {
       SQLConf.withExistingConf(c) {
         CatalystSqlParser.parseExpression(text)
       }
     } catch {
       case _: ParseException => CatalystSqlParser.parseExpression(text)
     }
+    // Neither the SQL path nor a temporary function of the session can shadow 
a built-in function.
+    StoredExpressionFunctions.qualifyBuiltinFunctions(parsed)
   }
 
   private def confWithSessionAnsi(session: SQLConf): SQLConf = {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
index 6161a1562409..918b7f3e6eee 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GeneratedColumnWriteSuite.scala
@@ -19,17 +19,29 @@ package org.apache.spark.sql.connector
 
 import java.util.Collections
 
+import 
test.org.apache.spark.sql.connector.catalog.functions.JavaStrLen.JavaStrLenStaticMagic
+
 import org.apache.spark.{SparkRuntimeException, SparkThrowable}
 import org.apache.spark.sql.{AnalysisException, DataFrame, QueryTest, Row}
-import org.apache.spark.sql.catalyst.QueryPlanningTracker
+import org.apache.spark.sql.catalyst.{InternalRow, QueryPlanningTracker}
 import org.apache.spark.sql.catalyst.expressions.CheckInvariant
 import org.apache.spark.sql.catalyst.util.GeneratedColumn
 import org.apache.spark.sql.connector.catalog.{Column, GenerationExpression, 
Identifier, InMemoryCatalog, InMemoryRowLevelOperationTableCatalog}
 import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.CatalogHelper
+import org.apache.spark.sql.connector.catalog.functions.{ScalarFunction, 
UnboundFunction}
 import org.apache.spark.sql.connector.expressions.{Expression => V2Expression, 
GeneralScalarExpression, Transform}
 import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
 import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{IntegerType, MetadataBuilder, StringType, 
StructType}
+import org.apache.spark.sql.types.{DataType, IntegerType, MetadataBuilder, 
StringType, StructType}
+import org.apache.spark.unsafe.types.UTF8String
+
+// A V2 function that a test binds to the name of a built-in function to 
shadow it.
+private object ConstantString extends ScalarFunction[UTF8String] {
+  override def inputTypes(): Array[DataType] = Array(StringType)
+  override def resultType(): DataType = StringType
+  override def name(): String = "constant_string"
+  override def produceResult(input: InternalRow): UTF8String = 
UTF8String.fromString("OK")
+}
 
 /**
  * Tests for generated column auto-fill and constraint enforcement during 
writes.
@@ -459,6 +471,69 @@ class GeneratedColumnWriteSuite extends QueryTest with 
DatasourceV2SQLBase {
     }
   }
 
+  private def addFunction(name: String, function: UnboundFunction): Unit = {
+    catalog("testcat").asInstanceOf[InMemoryCatalog]
+      .createFunction(Identifier.of(Array("ns"), name), function)
+  }
+
+  test("SPARK-60866: generation expression cannot call a V2 function with a 
magic method") {
+    val tblName = "my_tab"
+    Seq(
+      StrLenMagic -> "`strlen_magic`",
+      new JavaStrLenStaticMagic -> "`strlen`"
+    ).foreach { case (function, functionName) =>
+      addFunction("strlen", StrLen(function))
+      withTable(s"testcat.$tblName") {
+        checkError(
+          exception = intercept[AnalysisException] {
+            sql(s"""CREATE TABLE testcat.$tblName(
+                   |  s STRING,
+                   |  g INT GENERATED ALWAYS AS (testcat.ns.strlen(s))
+                   |) USING foo""".stripMargin)
+          },
+          condition = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+          parameters = Map(
+            "fieldName" -> "g",
+            "expressionStr" -> "testcat.ns.strlen(s)",
+            "reason" -> s"failed to resolve $functionName to a built-in 
function"))
+      }
+    }
+  }
+
+  test("SPARK-60866: SQL path does not shadow a built-in function of a 
generation expression") {
+    val tblName = "my_tab"
+    addFunction("upper", StrLen(ConstantString))
+    withTable(s"testcat.$tblName") {
+      sql(s"""CREATE TABLE testcat.$tblName(
+             |  s STRING,
+             |  g STRING GENERATED ALWAYS AS (upper(s))
+             |) USING foo""".stripMargin)
+      withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+        sql("SET PATH = testcat.ns, system.builtin")
+        try {
+          // The path does shadow the built-in function outside of the 
generation expression.
+          checkAnswer(sql("SELECT upper('x')"), Row("OK"))
+          sql(s"INSERT INTO testcat.$tblName(s) VALUES ('x')")
+          checkAnswer(spark.table(s"testcat.$tblName"), Row("x", "X"))
+          sql(s"INSERT INTO testcat.$tblName VALUES ('y', 'Y')")
+          checkError(
+            exception = intercept[SparkRuntimeException] {
+              sql(s"INSERT INTO testcat.$tblName VALUES ('z', 'OK')")
+            },
+            condition = "CHECK_CONSTRAINT_VIOLATION",
+            sqlState = "23001",
+            parameters = Map(
+              "constraintName" -> "Generated Column",
+              "expression" -> "g <=> upper(s)",
+              "values" -> " - g : OK"))
+          checkAnswer(spark.table(s"testcat.$tblName"), Seq(Row("x", "X"), 
Row("y", "Y")))
+        } finally {
+          sql("SET PATH = DEFAULT_PATH")
+        }
+      }
+    }
+  }
+
   test("INSERT by position does not auto-fill trailing generated column") {
     val tblName = "my_tab"
     withTable(s"testcat.$tblName") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to