This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 962b82fff650 [SPARK-52719][SQL] Support using scalar UDFs in TVF
arguments
962b82fff650 is described below
commit 962b82fff650d7bc7ed403c2f4a976811cb56a3a
Author: Anupam Yadav <[email protected]>
AuthorDate: Mon Jun 8 20:51:19 2026 -0700
[SPARK-52719][SQL] Support using scalar UDFs in TVF arguments
### What changes were proposed in this pull request?
Add a guard in `ResolveSQLFunctions` (Analyzer.scala) that allows
`SQLFunctionExpression` nodes inside `SQLTableFunction` inputs to pass through
for downstream resolution, instead of being eagerly rejected by the catch-all
case.
### Why are the changes needed?
`SELECT * FROM tvf(scalar_udf(true))` incorrectly throws
`AnalysisException: Using SQL function ... in SQLTableFunction is not
supported`. Scalar UDFs return scalar values and should be valid TVF arguments.
The catch-all case in `ResolveSQLFunctions` was firing on `SQLTableFunction`
before its inputs were resolved, incorrectly treating scalar UDFs as
unsupported.
Note: `SQLFunctionExpression` is exclusively for scalar functions -- TVFs
use `SQLTableFunction` (a LogicalPlan node). The guard cannot accidentally
permit nested TVFs because `SessionCatalog.makeSQLFunctionBuilder` throws
`NOT_A_SCALAR_FUNCTION` if a TVF name is used in a scalar position.
### Does this PR introduce _any_ user-facing change?
Yes -- scalar SQL UDFs can now be used as arguments to table-valued
functions.
### How was this patch tested?
Added 5 tests in `SQLFunctionSuite`:
1. **Happy path**: `tvf(scalar_udf(1))` returns correct result
2. **Negative test**: TVF-in-TVF argument is still rejected with
`NOT_A_SCALAR_FUNCTION`
3. **Nested scalar UDFs**: `tvf(add_one(add_one(1)))` resolves recursively
4. **Mixed args**: `multi_tvf(scalar_udf(3), 10)` with UDF + literal
5. **NULL propagation**: `tvf(null_udf(1))` correctly returns null
Without the fix, test 1 throws `AnalysisException`. With the fix, all 5
pass.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Closes #55950 from yadavay-amzn/fix/SPARK-52719-udf-in-tvf.
Authored-by: Anupam Yadav <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit cc64f0a3d84c9958e7b17fadafcf34d0612294a1)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../spark/sql/catalyst/analysis/Analyzer.scala | 6 ++
.../spark/sql/execution/SQLFunctionSuite.scala | 75 ++++++++++++++++++++++
2 files changed, 81 insertions(+)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 848bc92d6d73..a97e0fac6ce5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -2876,6 +2876,12 @@ class Analyzer(
j.copy(left = newLeft, right = newRight)
}
+ // Defer to a later iteration: once ResolveSQLTableFunctions expands the
+ // table function, the SQLFunctionExpression sits inside the expanded
+ // plan's input-cast Project and is rewritten by a subsequent iteration
+ // of this rule.
+ case f: SQLTableFunction if hasSQLFunctionExpression(f.expressions) => f
+
case o: LogicalPlan if o.resolved &&
hasSQLFunctionExpression(o.expressions) =>
o.transformExpressionsWithPruning(_.containsPattern(SQL_FUNCTION_EXPRESSION)) {
case f: SQLFunctionExpression =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLFunctionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLFunctionSuite.scala
index 4362064eb861..7406f953d063 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLFunctionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SQLFunctionSuite.scala
@@ -359,4 +359,79 @@ class SQLFunctionSuite extends SharedSparkSession {
}
}
}
+
+ test("SPARK-52719: scalar SQL UDF as TVF argument") {
+ withUserDefinedFunction("scalar_udf" -> false, "table_func" -> false) {
+ sql("CREATE FUNCTION scalar_udf(x INT) RETURNS INT RETURN x + 1")
+ sql(
+ """CREATE FUNCTION table_func(x INT)
+ |RETURNS TABLE(a INT)
+ |RETURN SELECT x
+ |""".stripMargin)
+ checkAnswer(sql("SELECT * FROM table_func(scalar_udf(1))"), Row(2))
+ }
+ }
+
+ test("SPARK-52719: table-valued SQL function in TVF argument is still
rejected") {
+ withUserDefinedFunction("tvf_inner" -> false, "tvf_outer" -> false) {
+ sql(
+ """CREATE FUNCTION tvf_inner(x INT)
+ |RETURNS TABLE(a INT)
+ |RETURN SELECT x
+ |""".stripMargin)
+ sql(
+ """CREATE FUNCTION tvf_outer(x INT)
+ |RETURNS TABLE(a INT)
+ |RETURN SELECT x
+ |""".stripMargin)
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql("SELECT * FROM tvf_outer(tvf_inner(1))").collect()
+ },
+ condition = "NOT_A_SCALAR_FUNCTION",
+ parameters = Map(
+ "functionName" -> "`spark_catalog`.`default`.`tvf_inner`"),
+ context = ExpectedContext(
+ fragment = "tvf_inner(1)",
+ start = 24,
+ stop = 35)
+ )
+ }
+ }
+
+ test("SPARK-52719: nested scalar UDFs in TVF argument") {
+ withUserDefinedFunction("add_one" -> false, "tvf" -> false) {
+ sql("CREATE FUNCTION add_one(x INT) RETURNS INT RETURN x + 1")
+ sql(
+ """CREATE FUNCTION tvf(x INT)
+ |RETURNS TABLE(a INT)
+ |RETURN SELECT x
+ |""".stripMargin)
+ checkAnswer(sql("SELECT * FROM tvf(add_one(add_one(1)))"), Row(3))
+ }
+ }
+
+ test("SPARK-52719: scalar UDF mixed with literals in TVF arguments") {
+ withUserDefinedFunction("add_one" -> false, "multi_tvf" -> false) {
+ sql("CREATE FUNCTION add_one(x INT) RETURNS INT RETURN x + 1")
+ sql(
+ """CREATE FUNCTION multi_tvf(x INT, y INT)
+ |RETURNS TABLE(a INT)
+ |RETURN SELECT x + y
+ |""".stripMargin)
+ checkAnswer(sql("SELECT * FROM multi_tvf(add_one(3), 10)"), Row(14))
+ }
+ }
+
+ test("SPARK-52719: scalar UDF returning NULL in TVF argument") {
+ withUserDefinedFunction("null_udf" -> false, "tvf" -> false) {
+ sql("CREATE FUNCTION null_udf(x INT) RETURNS INT RETURN CAST(NULL AS
INT)")
+ sql(
+ """CREATE FUNCTION tvf(x INT)
+ |RETURNS TABLE(a INT)
+ |RETURN SELECT x
+ |""".stripMargin)
+ checkAnswer(sql("SELECT * FROM tvf(null_udf(1))"), Row(null))
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
