This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 84f97adbd04c [SPARK-60447][SQL] Support V2 functions in CREATE
FUNCTION on session catalog
84f97adbd04c is described below
commit 84f97adbd04c7a4d7e1be5ed2f0076fde2cd5d9a
Author: Cheng Pan <[email protected]>
AuthorDate: Tue Nov 3 10:11:15 2026 +0800
[SPARK-60447][SQL] Support V2 functions in CREATE FUNCTION on session
catalog
### What changes were proposed in this pull request?
This PR allows `CREATE [TEMPORARY] FUNCTION ... AS 'class' [USING JAR ...]`
on the session
catalog to register a DataSource V2 function, i.e. a class implementing
`org.apache.spark.sql.connector.catalog.functions.UnboundFunction`, the
same way as Hive UDFs.
```sql
ADD JAR /path/to/my-udfs.jar;
CREATE TEMPORARY FUNCTION strlen AS 'com.example.StrLen';
-- or persistent, with the jar loaded on first use
CREATE FUNCTION strlen AS 'com.example.StrLen' USING JAR
'hdfs:///udfs/my-udfs.jar';
SELECT strlen(name) FROM t;
```
Persistent functions are stored in the external catalog in the same shape
as Hive UDFs (real
class name and resource URIs), with no new metadata format; the function
kind is decided by the
class when loaded. They behave like Hive UDFs: a missing class, an
inaccessible jar, or a class
that cannot be instantiated only fails when the function is invoked, not on
CREATE, SHOW,
DESCRIBE or listing, which do not initialize the class of a persistent
function. Column names use
the function name. HMS calls are the same as for a Hive UDF.
How other engines sharing the HMS react to such an entry:
1. Hive: an entry it cannot register is logged and skipped.
2. Flink: lists it, and fails only when it is called.
3. Impala: skips it with an error log.
4. Trino, Doris, StarRocks: do not read HMS function entries created by
other engines.
Flink and Impala store their own Java functions in this same shape. A class
implementing both a
Hive UDF interface and `UnboundFunction` runs as a V2 function in Spark and
as a Hive UDF in
other engines.
### Known limitation
A scalar V2 function that implements the magic method (`invoke`), and whose
jar is only added
at runtime (`ADD JAR`, or `CREATE FUNCTION ... USING JAR`), may fail to
compile in generated code
when the context classloader of the driver thread running the query cannot
see the jar. For
example:
1. In spark-shell, the jar is loaded by one line, and a later line calls
the function, since the
REPL resets the context classloader after each line.
2. In a multi-threaded driver application, a thread other than the one that
loaded the jar runs
the query.
3. Spark Connect enabled via `spark.plugins` without
`spark.sql.artifact.isolation.alwaysApplyClassloader=true`, where
requests run on different
threads.
With default configs, whole-stage codegen falls back with a warning and the
query returns correct
results more slowly, but operators without a fallback fail the query, e.g.
`SELECT c FROM t ORDER BY f(c) LIMIT 10`.
Not affected, for the query's main thread:
1. Spark Thrift Server, which sets the jar classloader for each operation.
2. Spark Connect server started by `start-connect-server.sh`, which applies
the session
classloader to each request.
3. Functions implementing `produceResult` only, and aggregate functions.
4. Jars provided at startup via `--jars` / `spark.jars`.
This is a pre-existing issue for V2 functions served by a `FunctionCatalog`
from runtime-added
jars, tracked by
[SPARK-60811](https://issues.apache.org/jira/browse/SPARK-60811); this feature
makes it easier to trigger.
### Why are the changes needed?
The V2 function API is the modern, codegen-friendly way to write Spark
functions, but it is only
usable through a custom `FunctionCatalog`. This PR lets session catalog
users, including HMS,
use V2 functions with the familiar `ADD JAR` / `CREATE FUNCTION` workflow,
without implementing
a catalog plugin.
### Does this PR introduce _any_ user-facing change?
Yes.
1. `CREATE [TEMPORARY] FUNCTION ... AS 'class'` accepts a class
implementing `UnboundFunction`.
Previously, calling such a function failed with `NO_HANDLER_FOR_UDAF`,
or with
`_LEGACY_ERROR_TEMP_2450` when Hive support is enabled.
2. `DESCRIBE FUNCTION` on such a function shows the function's description;
for a persistent
function, once it has been invoked in the session.
3. A class implementing `UnboundFunction` that also extends a Hive UDF
class or
`UserDefinedAggregateFunction` now runs as a V2 function, including
functions created in
earlier versions. See the migration guide.
4. New error condition `CANNOT_INSTANTIATE_FUNCTION_CLASS`, raised when
such a class cannot be
instantiated.
### How was this patch tested?
New `CreateV2FunctionSuite`, and a new test in `HiveUDFSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59697 from pan3793/v2-udf.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 7640b3df675d479e3f41ac2bc33dcccf0620c015)
Signed-off-by: Cheng Pan <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 6 +
docs/sql-migration-guide.md | 2 +-
docs/sql-ref-syntax-ddl-create-function.md | 11 +-
.../sql/catalyst/expressions/ExpressionInfo.java | 2 +-
.../sql/catalyst/analysis/FunctionResolution.scala | 86 +--
.../sql/catalyst/catalog/SessionCatalog.scala | 104 +++-
.../catalyst/expressions/V2ExpressionUtils.scala | 54 +-
.../apache/spark/sql/catalyst/util/package.scala | 7 +-
.../spark/sql/errors/QueryCompilationErrors.scala | 27 +-
.../catalog/functions/JavaInitTrackingStrLen.java | 66 +++
.../functions/JavaStaticInitFailingStrLen.java | 61 ++
.../sql/connector/CreateV2FunctionSuite.scala | 635 +++++++++++++++++++++
.../sql/expressions/ExpressionInfoSuite.scala | 2 +-
.../spark/sql/hive/execution/HiveUDFSuite.scala | 35 +-
14 files changed, 997 insertions(+), 101 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index cf0c93a07f0d..56a246f7a05c 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -641,6 +641,12 @@
],
"sqlState" : "22546"
},
+ "CANNOT_INSTANTIATE_FUNCTION_CLASS" : {
+ "message" : [
+ "Cannot instantiate class <className> when registering the function
<functionName>: <msg>. Please make sure the class has a public no-arg
constructor that succeeds."
+ ],
+ "sqlState" : "46103"
+ },
"CANNOT_INSTANTIATE_HIVE_FUNCTION" : {
"message" : [
"Cannot instantiate the Hive UDF/UDAF/UDTF wrapper class <clazz>. Check
that the function arguments and their types are supported."
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 30bbaf7bc163..ebf71c04ce4b 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -63,7 +63,6 @@ license: |
- Since Spark 4.4, when `spark.sql.parquet.fieldId.read.enabled` is true, a
Parquet read fails with `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE` if two or more
requested fields of a struct (or of the top-level schema) resolve to the same
Parquet field. This happens when one requested field is matched by field ID and
another by name, or when two requested fields have the same field ID. In Spark
4.3 and earlier, such a read returned wrong values without an error: with the
vectorized reader one [...]
- Since Spark 4.4, when `spark.sql.groupByOrdinal` is true (the default), the
pipe `AGGREGATE` operator rejects an aliased integer literal in its `GROUP BY`
clause, such as `TABLE t |> AGGREGATE sum(x) AS s GROUP BY 2 AS grp`, with
`UNSUPPORTED_FEATURE.PIPE_OPERATOR_AGGREGATE_UNSUPPORTED_CASE`, because a bare
integer literal in the same position is an ordinal. Previously the aliased
literal was grouped as a constant. Bare ordinals and aliases on other grouping
expressions are not affecte [...]
- Since Spark 4.4, with whole-stage codegen enabled, a filter condition,
including the `FILTER` clause of an aggregate function, evaluates the remaining
top-level conjuncts of the condition on a row after one of them evaluates to
`NULL`, as the SQL standard requires, and does so in the order they appear in
the condition, as Spark already did with whole-stage codegen disabled, which
also evaluates a subexpression shared by several conjuncts before the
condition. Only a `FALSE` conjunct st [...]
-
- Since Spark 4.4, when a format is specified, `to_timestamp`,
`try_to_timestamp`, and `to_timestamp_ltz` cast a numeric first argument to a
string and parse it with the format, for example `to_timestamp(20201231,
'yyyyMMdd')` returns `2020-12-31 00:00:00`. This matches the existing behavior
when `spark.sql.timestampType` is `TIMESTAMP_NTZ`. Previously, with the default
`TIMESTAMP_LTZ` timestamp type, such calls failed with `INTERNAL_ERROR`. Note
that the string form of the number is par [...]
- Since Spark 4.4, the Spark Thrift Server logs a warning at startup when
`hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies
user identities (any recognized type other than `NONE`/`NOSASL`), because
executor-side data access still runs as the server's own service identity
rather than the connecting user (SPARK-5159). Set
`spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true` to acknowledge the
limitation and silence the warning. Spark 5.0 is expected to [...]
- Since Spark 4.4, `UPDATE` and `MERGE` on a Data Source V2 table fail with
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` when they assign to
a metadata column of the table. This includes `UPDATE SET *` and `INSERT *` in
`MERGE WITH SCHEMA EVOLUTION` when the source has a column with the name of a
metadata column of the target table. Previously, these assignments passed
analysis and were usually ignored. To restore the previous behavior, set
`spark.sql.legacy.ignoreMetadata [...]
@@ -71,6 +70,7 @@ license: |
- Since Spark 4.4, `CREATE TABLE` and `REPLACE TABLE` on a Data Source V2
table fail with `AGGREGATE_OR_WINDOW_FUNCTION_IN_CHECK_CONSTRAINT` when the
condition of a CHECK constraint calls an aggregate or window function,
including a user-defined aggregate function, `grouping`, and `grouping_id`.
`ALTER TABLE ... ADD CONSTRAINT` keeps failing with its existing errors, such
as `INVALID_WHERE_CONDITION` and `WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE`.
Previously, these statements succeeded and [...]
- Since Spark 4.4, a write to a Data Source V2 table that parses the SQL text
of a CHECK constraint binds an unqualified built-in function name to
`system.builtin`, regardless of the SQL path and
`spark.sql.functionResolution.sessionOrder` (SPARK-60777). Previously, an
existing constraint could call another function with the same name.
- Since Spark 4.4, `CREATE TABLE`, `REPLACE TABLE`, and `ALTER TABLE ... ADD
CONSTRAINT` on a Data Source V2 table fail with `SUBQUERY_IN_CHECK_CONSTRAINT`
when the condition of a CHECK constraint contains a subquery. `ALTER TABLE ...
ADD CONSTRAINT` keeps failing with
`UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.HIGHER_ORDER_FUNCTION` on a subquery
in a lambda function. Previously, most of these statements failed with
`INTERNAL_ERROR`, but a subquery that needed no resolution, such as `id [...]
+- Since Spark 4.4, a function created by `CREATE FUNCTION ... AS 'class'`
whose class implements
`org.apache.spark.sql.connector.catalog.functions.UnboundFunction` and extends
either a Hive UDF class (such as `UDF` or `GenericUDF`) or
`org.apache.spark.sql.expressions.UserDefinedAggregateFunction` runs as a
DataSource V2 function instead of a Hive UDF or Spark UDAF, including functions
created in earlier versions. Its result type, argument coercion, and null
handling are defined by the V [...]
## Upgrading from Spark SQL 4.2 to 4.3
diff --git a/docs/sql-ref-syntax-ddl-create-function.md
b/docs/sql-ref-syntax-ddl-create-function.md
index 256587049441..3367d539a0c3 100644
--- a/docs/sql-ref-syntax-ddl-create-function.md
+++ b/docs/sql-ref-syntax-ddl-create-function.md
@@ -80,13 +80,22 @@ CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [ IF NOT
EXISTS ]
* **class_name**
Specifies the name of the class that provides the implementation for
function to be created.
- The implementing class should extend one of the base classes as follows:
+ The implementing class should have a public no-arg constructor, and extend
one of the base
+ classes as follows:
+ * Should implement `UnboundFunction` in
`org.apache.spark.sql.connector.catalog.functions`
+ package.
* Should extend `UDF` or `UDAF` in `org.apache.hadoop.hive.ql.exec`
package.
* Should extend `AbstractGenericUDAFResolver`, `GenericUDF`, or
`GenericUDTF` in `org.apache.hadoop.hive.ql.udf.generic` package.
* Should extend `UserDefinedAggregateFunction` in
`org.apache.spark.sql.expressions` package.
+ **Note:** If an `UnboundFunction` binds to a `ScalarFunction` that
implements the magic method
+ `invoke`, and its jar is added at runtime (`ADD JAR`, or `USING JAR`),
codegen may fail to find
+ the class, e.g., in spark-shell. Whole-stage codegen falls back to slower
interpreted
+ execution, other codegen fails the query. To avoid it, add the jar at
startup via `--jars`, or
+ implement `produceResult` instead.
+
* **resource_locations**
Specifies the list of resources that contain the implementation of the
function
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java
index 4fd6e543cf09..89b65422b186 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java
@@ -53,7 +53,7 @@ public class ExpressionInfo {
private static final Set<String> validSources =
new HashSet<>(Arrays.asList("built-in", "hive", "python_udf",
"scala_udf", "sql_udf",
- "java_udf", "python_udtf", "internal"));
+ "java_udf", "python_udtf", "internal", "v2_function"));
public String getClassName() {
return className;
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
index ebaecdb117e8..9bb099b13c09 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala
@@ -54,11 +54,7 @@ object FunctionType {
case object NotFound extends FunctionType
}
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-import org.apache.spark.sql.connector.catalog.functions.{
- AggregateFunction => V2AggregateFunction,
- ScalarFunction,
- UnboundFunction
-}
+import org.apache.spark.sql.connector.catalog.functions.UnboundFunction
import org.apache.spark.sql.errors.{DataTypeErrorsBase, QueryCompilationErrors}
import org.apache.spark.sql.internal.connector.V1Function
import org.apache.spark.sql.types._
@@ -686,7 +682,7 @@ class FunctionResolution(
// clauses (DISTINCT / FILTER / ORDER BY / IGNORE NULLS). They have
dedicated physical
// operators that do not honor them, so reject rather than silently
drop the clause.
if (agg.isInstanceOf[PythonUDAF] || agg.isInstanceOf[PythonAggregate])
{
- checkUnsupportedAggregateClause(agg, unresolvedFunc)
+ checkUnsupportedAggregateClause(agg.prettyName, unresolvedFunc)
}
// After parse, the functions not set the ordering within group yet.
val newAgg = agg match {
@@ -718,7 +714,7 @@ class FunctionResolution(
aggFunc.toAggregateExpression(unresolvedFunc.isDistinct,
unresolvedFunc.filter)
// This function is not an aggregate function, just return the resolved
one.
case other =>
- checkUnsupportedAggregateClause(other, unresolvedFunc)
+ checkUnsupportedAggregateClause(other.prettyName, unresolvedFunc)
if (other.isInstanceOf[String2TrimExpression] && numArgs == 2) {
if (trimWarningEnabled.get) {
log.warn(
@@ -734,21 +730,21 @@ class FunctionResolution(
}
private def checkUnsupportedAggregateClause(
- func: Expression,
+ funcName: String,
unresolvedFunc: UnresolvedFunction): Unit = {
if (unresolvedFunc.isDistinct) {
- throw
QueryCompilationErrors.functionWithUnsupportedSyntaxError(func.prettyName,
"DISTINCT")
+ throw
QueryCompilationErrors.functionWithUnsupportedSyntaxError(funcName, "DISTINCT")
}
if (unresolvedFunc.filter.isDefined) {
throw QueryCompilationErrors.functionWithUnsupportedSyntaxError(
- func.prettyName,
+ funcName,
"FILTER clause"
)
}
// Only fail for IGNORE NULLS; RESPECT NULLS is the default behavior
if (unresolvedFunc.ignoreNulls.contains(true)) {
throw QueryCompilationErrors.functionWithUnsupportedSyntaxError(
- func.prettyName,
+ funcName,
"IGNORE NULLS"
)
}
@@ -796,80 +792,28 @@ class FunctionResolution(
unbound: UnboundFunction,
arguments: Seq[Expression],
unresolvedFunc: UnresolvedFunction): Expression = {
- val inputType = StructType(arguments.zipWithIndex.map {
- case (exp, pos) => StructField(s"_$pos", exp.dataType, exp.nullable)
- })
- val bound = try {
- unbound.bind(inputType)
- } catch {
- case unsupported: UnsupportedOperationException =>
- throw QueryCompilationErrors.functionCannotProcessInputError(
- unbound,
- arguments,
- unsupported
- )
- }
-
- if (bound.inputTypes().length != arguments.length) {
- throw
QueryCompilationErrors.v2FunctionInvalidInputTypeLengthError(bound, arguments)
- }
-
- bound match {
- case scalarFunc: ScalarFunction[_] =>
- processV2ScalarFunction(scalarFunc, arguments, unresolvedFunc)
- case aggFunc: V2AggregateFunction[_, _] =>
- processV2AggregateFunction(aggFunc, arguments, unresolvedFunc)
- case _ =>
- failAnalysis(
- errorClass = "INVALID_UDF_IMPLEMENTATION",
- messageParameters = Map("funcName" -> toSQLId(bound.name()))
- )
- }
- }
-
- private def processV2ScalarFunction(
- scalarFunc: ScalarFunction[_],
- arguments: Seq[Expression],
- unresolvedFunc: UnresolvedFunction): Expression = {
- if (unresolvedFunc.isDistinct) {
- throw
QueryCompilationErrors.functionWithUnsupportedSyntaxError(scalarFunc.name(),
"DISTINCT")
- } else if (unresolvedFunc.filter.isDefined) {
- throw QueryCompilationErrors.functionWithUnsupportedSyntaxError(
- scalarFunc.name(),
- "FILTER clause"
- )
- } else if (unresolvedFunc.ignoreNulls.contains(true)) {
- // Only fail for IGNORE NULLS; RESPECT NULLS is the default behavior
- throw QueryCompilationErrors.functionWithUnsupportedSyntaxError(
- scalarFunc.name(),
- "IGNORE NULLS"
- )
- } else {
- V2ExpressionUtils.resolveScalarFunction(scalarFunc, arguments)
+ V2ExpressionUtils.bindFunction(unbound, arguments) match {
+ case aggregator: V2Aggregator[_, _] =>
+ processV2AggregateFunction(aggregator, unresolvedFunc)
+ case scalarExpr =>
+ checkUnsupportedAggregateClause(unbound.name(), unresolvedFunc)
+ scalarExpr
}
}
private def processV2AggregateFunction(
- aggFunc: V2AggregateFunction[_, _],
- arguments: Seq[Expression],
+ aggregator: V2Aggregator[_, _],
unresolvedFunc: UnresolvedFunction): Expression = {
// Only fail for IGNORE NULLS; RESPECT NULLS is the default behavior
if (unresolvedFunc.ignoreNulls.contains(true)) {
throw QueryCompilationErrors.functionWithUnsupportedSyntaxError(
- aggFunc.name(),
+ aggregator.aggrFunc.name(),
"IGNORE NULLS"
)
}
- val aggregator = V2Aggregator(aggFunc, arguments)
aggregator.toAggregateExpression(unresolvedFunc.isDistinct,
unresolvedFunc.filter)
}
- private def failAnalysis(errorClass: String, messageParameters: Map[String,
String]): Nothing = {
- throw new AnalysisException(
- errorClass = errorClass,
- messageParameters = messageParameters)
- }
-
/**
* Resolves [[UnresolvedProcedure]] for `CALL` / `DESCRIBE PROCEDURE` using
the same multipart
* candidates as SQL functions and relations ([[resolutionCandidates]] /
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
index 96a36ef59e15..5c58eef2c72e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
@@ -17,12 +17,14 @@
package org.apache.spark.sql.catalyst.catalog
+import java.lang.reflect.InvocationTargetException
import java.net.URI
import java.util.Locale
import java.util.concurrent.{Callable, ExecutionException, TimeUnit}
import javax.annotation.concurrent.GuardedBy
import scala.collection.mutable
+import scala.util.control.NonFatal
import com.google.common.cache.{Cache, CacheBuilder}
import com.google.common.util.concurrent.UncheckedExecutionException
@@ -40,6 +42,7 @@ import
org.apache.spark.sql.catalyst.catalog.SQLFunction.{padArgumentsWithDefaul
import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute,
AttributeReference, Cast, Expression, ExpressionInfo, LateralSubquery,
NamedArgumentExpression, NamedExpression, OuterReference, ScalarSubquery,
UpCast}
import org.apache.spark.sql.catalyst.expressions.NamedLambdaVariable
import org.apache.spark.sql.catalyst.expressions.UnresolvedNamedLambdaVariable
+import org.apache.spark.sql.catalyst.expressions.V2ExpressionUtils
import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser,
ParserInterface}
import org.apache.spark.sql.catalyst.plans.Inner
import org.apache.spark.sql.catalyst.plans.logical.{FunctionSignature,
InputParameter, LateralJoin, LogicalPlan, NamedParametersSupport,
OneRowRelation, Project, SubqueryAlias, View}
@@ -47,6 +50,7 @@ import org.apache.spark.sql.catalyst.trees.CurrentOrigin
import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, StringUtils}
import org.apache.spark.sql.connector.catalog.CatalogManager
import
org.apache.spark.sql.connector.catalog.CatalogManager.SESSION_CATALOG_NAME
+import org.apache.spark.sql.connector.catalog.functions.UnboundFunction
import org.apache.spark.sql.errors.{QueryCompilationErrors,
QueryExecutionErrors}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.StaticSQLConf.GLOBAL_TEMP_DATABASE
@@ -2073,17 +2077,69 @@ class SessionCatalog(
}
/**
- * Constructs a [[FunctionBuilder]] based on the provided function metadata.
+ * Constructs a [[FunctionBuilder]] and the [[ExpressionInfo]] based on the
provided function
+ * metadata.
*/
- private def makeFunctionBuilder(func: CatalogFunction): FunctionBuilder = {
+ private def makeFunctionBuilderAndInfo(
+ func: CatalogFunction): (FunctionBuilder, ExpressionInfo) = {
+ val clazz = loadFunctionClass(func)
+ val name = func.identifier.unquotedString
+ if (isV2FunctionClass(clazz)) {
+ // A V2 function: bind it to the arguments at each call site. An
aggregate function is
+ // returned as a bare `V2Aggregator`, which the caller wraps into an
`AggregateExpression`
+ // carrying DISTINCT / FILTER. This check goes first so that a class
also implementing a
+ // Hive UDF interface, for other engines sharing the metastore, runs as
a V2 function in
+ // Spark. The alias makes errors and column names use the function name,
like the Hive UDF
+ // wrappers do.
+ val unbound = newV2FunctionInstance(clazz, func.identifier)
+ val builder = (input: Seq[Expression]) => {
+ val expr = V2ExpressionUtils.bindFunction(
+ unbound, input, displayName = Some(func.identifier))
+ expr.setTagValue(FunctionRegistry.FUNC_ALIAS, name)
+ expr
+ }
+ (builder, makeExprInfoForV2Function(func, unbound))
+ } else {
+ val builder = (input: Seq[Expression]) =>
+ functionExpressionBuilder.makeExpression(name, clazz, input)
+ (builder, makeExprInfoForHiveFunction(func))
+ }
+ }
+
+ /**
+ * Loads the class a class-based function (`CREATE FUNCTION ... AS 'class'`)
refers to.
+ */
+ private def loadFunctionClass(func: CatalogFunction): Class[_] = {
val className = func.className
if (!Utils.classIsLoadable(className)) {
throw QueryCompilationErrors.cannotLoadClassWhenRegisteringFunctionError(
className, func.identifier)
}
- val clazz = Utils.classForName(className)
- val name = func.identifier.unquotedString
- (input: Seq[Expression]) => functionExpressionBuilder.makeExpression(name,
clazz, input)
+ Utils.classForName(className)
+ }
+
+ private def isV2FunctionClass(clazz: Class[_]): Boolean = {
+ classOf[UnboundFunction].isAssignableFrom(clazz)
+ }
+
+ private def newV2FunctionInstance(
+ clazz: Class[_],
+ ident: FunctionIdentifier): UnboundFunction = {
+ try {
+
clazz.getDeclaredConstructor().newInstance().asInstanceOf[UnboundFunction]
+ } catch {
+ case e: InvocationTargetException =>
+ val cause = e.getCause
+ if (NonFatal(cause) || cause.isInstanceOf[LinkageError]) {
+ throw QueryCompilationErrors.cannotInstantiateFunctionClassError(
+ clazz.getName, ident, cause)
+ } else {
+ throw cause
+ }
+ case NonFatal(e) =>
+ throw QueryCompilationErrors.cannotInstantiateFunctionClassError(
+ clazz.getName, ident, e)
+ }
}
private def makeUserDefinedScalarFuncBuilder(func: UserDefinedFunction):
FunctionBuilder = {
@@ -2119,8 +2175,13 @@ class SessionCatalog(
funcDefinition: CatalogFunction,
overrideIfExists: Boolean,
functionBuilder: Option[FunctionBuilder] = None): Unit = {
- val builder =
functionBuilder.getOrElse(makeFunctionBuilder(funcDefinition))
- registerFunction(funcDefinition, overrideIfExists, functionRegistry,
builder, info = None)
+ functionBuilder match {
+ case Some(builder) =>
+ registerFunction(funcDefinition, overrideIfExists, functionRegistry,
builder, info = None)
+ case None =>
+ val (builder, info) = makeFunctionBuilderAndInfo(funcDefinition)
+ registerFunction(funcDefinition, overrideIfExists, functionRegistry,
builder, Some(info))
+ }
}
private def registerFunction[T](
@@ -2173,6 +2234,28 @@ class SessionCatalog(
registry.registerFunction(identToRegister, resolvedInfo, functionBuilder)
}
+ private def makeExprInfoForV2Function(
+ func: CatalogFunction,
+ unbound: UnboundFunction): ExpressionInfo = {
+ val description = try {
+ unbound.description()
+ } catch {
+ case NonFatal(_) | _: LinkageError => null
+ }
+ new ExpressionInfo(
+ func.className,
+ func.identifier.database.orNull,
+ func.identifier.funcName,
+ description,
+ "",
+ "",
+ "",
+ "",
+ "",
+ "",
+ "v2_function")
+ }
+
private def makeExprInfoForHiveFunction(func: CatalogFunction):
ExpressionInfo = {
new ExpressionInfo(
func.className,
@@ -2736,12 +2819,7 @@ class SessionCatalog(
makeUserDefinedScalarFuncBuilder(udf))
} else {
loadFunctionResources(funcMetadata.resources)
- registerFunction(
- funcMetadata,
- overrideIfExists = false,
- functionRegistry,
- makeFunctionBuilder(funcMetadata),
- info = None)
+ registerFunction(funcMetadata, overrideIfExists = false)
}
functionRegistry.lookupFunctionBuilder(qualifiedIdent).get
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala
index 2ca5a499b84e..8e1e2be16a9a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala
@@ -22,9 +22,10 @@ import java.lang.reflect.{Method, Modifier}
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys.{FUNCTION_NAME, FUNCTION_PARAM}
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.{InternalRow, SQLConfHelper}
+import org.apache.spark.sql.catalyst.{FunctionIdentifier, InternalRow,
SQLConfHelper}
import org.apache.spark.sql.catalyst.analysis.{NoSuchFunctionException,
UnresolvedAttribute}
import org.apache.spark.sql.catalyst.encoders.EncoderUtils
+import org.apache.spark.sql.catalyst.expressions.aggregate.V2Aggregator
import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation,
LogicalPlan, SampleMethod}
import org.apache.spark.sql.catalyst.types.DataTypeUtils.fromAttributes
@@ -210,9 +211,54 @@ object V2ExpressionUtils extends SQLConfHelper with
Logging {
case V2SampleMethod.SYSTEM => SampleMethod.System
}
+ /**
+ * Binds a V2 [[UnboundFunction]] to the given arguments and turns the
resulting
+ * [[BoundFunction]] into a Catalyst expression. The optional display name
preserves identifier
+ * parts in diagnostics.
+ *
+ * An aggregate function is returned as a bare [[V2Aggregator]] so that
callers can wrap it into
+ * an `AggregateExpression` carrying DISTINCT / FILTER, which is not known
here.
+ */
+ def bindFunction(
+ unbound: UnboundFunction,
+ arguments: Seq[Expression],
+ displayName: Option[FunctionIdentifier] = None): Expression = {
+ val inputType = StructType(arguments.zipWithIndex.map {
+ case (exp, pos) => StructField(s"_$pos", exp.dataType, exp.nullable)
+ })
+ val bound = try {
+ unbound.bind(inputType)
+ } catch {
+ case unsupported: UnsupportedOperationException =>
+ throw QueryCompilationErrors.functionCannotProcessInputError(
+ displayName.map(_.unquotedString).getOrElse(unbound.name()),
arguments, unsupported)
+ }
+
+ val inputTypesLength = bound.inputTypes().length
+ if (inputTypesLength != arguments.length) {
+ throw QueryCompilationErrors.v2FunctionInvalidInputTypeLengthError(
+ displayName.map(_.unquotedString).getOrElse(bound.name()),
+ arguments.length,
+ inputTypesLength)
+ }
+
+ bound match {
+ case scalarFunc: ScalarFunction[_] =>
+ resolveScalarFunction(scalarFunc, arguments, displayName)
+ case aggFunc: AggregateFunction[_, _] =>
+ V2Aggregator(aggFunc, arguments)
+ case _ =>
+ throw new AnalysisException(
+ errorClass = "INVALID_UDF_IMPLEMENTATION",
+ messageParameters = Map(
+ "funcName" ->
displayName.map(_.quotedString).getOrElse(toSQLId(bound.name()))))
+ }
+ }
+
def resolveScalarFunction(
scalarFunc: ScalarFunction[_],
- arguments: Seq[Expression]): Expression = {
+ arguments: Seq[Expression],
+ displayName: Option[FunctionIdentifier] = None): Expression = {
val declaredInputTypes = scalarFunc.inputTypes().toImmutableArraySeq
val argClasses = declaredInputTypes.map(EncoderUtils.dataTypeJavaClass)
findMethod(scalarFunc, MAGIC_METHOD_NAME, argClasses) match {
@@ -238,7 +284,9 @@ object V2ExpressionUtils extends SQLConfHelper with Logging
{
case _ =>
throw new AnalysisException(
errorClass = "SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED",
- messageParameters = Map("scalarFunc" ->
toSQLId(scalarFunc.name())))
+ messageParameters = Map(
+ "scalarFunc" -> displayName.map(_.quotedString).getOrElse(
+ toSQLId(scalarFunc.name()))))
}
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala
index 77e8dcc7d127..5842282c9010 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala
@@ -21,8 +21,9 @@ import java.io._
import java.nio.charset.StandardCharsets.UTF_8
import org.apache.spark.internal.Logging
-import org.apache.spark.sql.catalyst.analysis.TempResolvedColumn
+import org.apache.spark.sql.catalyst.analysis.{FunctionRegistry,
TempResolvedColumn}
import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.objects.InvokeLike
import
org.apache.spark.sql.catalyst.util.FieldMetadataUtils.FIELD_ID_METADATA_KEY
import org.apache.spark.sql.connector.catalog.MetadataColumn
import org.apache.spark.sql.types.{MetadataBuilder, NumericType, StringType,
StructType}
@@ -136,6 +137,10 @@ package object util extends Logging {
)
case c: Cast if !c.containsTag(Cast.USER_SPECIFIED_CAST) =>
PrettyAttribute(usePrettyExpression(c.child,
shouldTrimTempResolvedColumn).sql, c.dataType)
+ // A V2 function's magic method registered under a function name, e.g. by
`CREATE FUNCTION`.
+ case i: InvokeLike if i.getTagValue(FunctionRegistry.FUNC_ALIAS).isDefined
=>
+ val args = i.arguments.map(toPrettySQL(_, shouldTrimTempResolvedColumn))
+ PrettyAttribute(s"${i.prettyName}(${args.mkString(", ")})", i.dataType)
case j: RoutedSqlJsonExpression =>
// Column names are for display and never reparsed, so render the clean
clause-free form, not
// the canonical `sql` that appends a round-trip-only default clause.
Render children in place
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
index 181398ca1590..4af1c7d1a13d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
@@ -35,7 +35,6 @@ import org.apache.spark.sql.catalyst.trees.{Origin, TreeNode}
import org.apache.spark.sql.catalyst.util.{quoteIdentifier, ParseMode}
import org.apache.spark.sql.connector.catalog._
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-import org.apache.spark.sql.connector.catalog.functions.{BoundFunction,
UnboundFunction}
import org.apache.spark.sql.connector.expressions.filter.Predicate
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.LEGACY_CTE_PRECEDENCE_POLICY
@@ -1448,6 +1447,17 @@ private[sql] object QueryCompilationErrors extends
QueryErrorsBase with Compilat
new FunctionAlreadyExistsException(func.nameParts)
}
+ def cannotInstantiateFunctionClassError(
+ className: String, func: FunctionIdentifier, cause: Throwable):
Throwable = {
+ new AnalysisException(
+ errorClass = "CANNOT_INSTANTIATE_FUNCTION_CLASS",
+ messageParameters = Map(
+ "className" -> className,
+ "functionName" -> toSQLId(func.toString),
+ "msg" -> Option(cause.getMessage).getOrElse(cause.getClass.getName)),
+ cause = Some(cause))
+ }
+
def cannotLoadClassWhenRegisteringFunctionError(
className: String, func: FunctionIdentifier): Throwable = {
new AnalysisException(
@@ -2794,27 +2804,28 @@ private[sql] object QueryCompilationErrors extends
QueryErrorsBase with Compilat
}
def functionCannotProcessInputError(
- unbound: UnboundFunction,
+ functionName: String,
arguments: Seq[Expression],
unsupported: UnsupportedOperationException): Throwable = {
new AnalysisException(
errorClass = "_LEGACY_ERROR_TEMP_1198",
messageParameters = Map(
- "unbound" -> unbound.name,
+ "unbound" -> functionName,
"arguments" -> arguments.map(_.dataType.simpleString).mkString(", "),
"unsupported" -> unsupported.getMessage),
cause = Some(unsupported))
}
def v2FunctionInvalidInputTypeLengthError(
- bound: BoundFunction,
- args: Seq[Expression]): Throwable = {
+ functionName: String,
+ argsLength: Int,
+ inputTypesLength: Int): Throwable = {
new AnalysisException(
errorClass = "_LEGACY_ERROR_TEMP_1199",
messageParameters = Map(
- "bound" -> bound.name(),
- "argsLen" -> args.length.toString,
- "inputTypesLen" -> bound.inputTypes().length.toString))
+ "bound" -> functionName,
+ "argsLen" -> argsLength.toString,
+ "inputTypesLen" -> inputTypesLength.toString))
}
def cannotResolveColumnNameAmongAttributesError(
diff --git
a/sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaInitTrackingStrLen.java
b/sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaInitTrackingStrLen.java
new file mode 100644
index 000000000000..778346832183
--- /dev/null
+++
b/sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaInitTrackingStrLen.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package test.org.apache.spark.sql.connector.catalog.functions;
+
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction;
+import org.apache.spark.sql.connector.catalog.functions.SimpleFunction;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+
+/**
+ * A string length function whose class initialization requires the class
+ * `test.v2udf.UsingResourceStrLen`, which only exists in the function's
`USING` jar, and records
+ * the initialization in a system property.
+ */
+public class JavaInitTrackingStrLen implements SimpleFunction,
ScalarFunction<Integer> {
+ static {
+ try {
+ Class.forName(
+ "test.v2udf.UsingResourceStrLen", true,
Thread.currentThread().getContextClassLoader());
+ } catch (ClassNotFoundException e) {
+ throw new IllegalStateException(e);
+ }
+ System.setProperty("spark.test.JavaInitTrackingStrLen.initialized",
"true");
+ }
+
+ @Override
+ public DataType[] inputTypes() {
+ return new DataType[] { DataTypes.StringType };
+ }
+
+ @Override
+ public DataType resultType() {
+ return DataTypes.IntegerType;
+ }
+
+ @Override
+ public String name() {
+ return "strlen";
+ }
+
+ @Override
+ public String description() {
+ return "strlen(string) -> int";
+ }
+
+ @Override
+ public Integer produceResult(InternalRow input) {
+ return input.getUTF8String(0).numChars();
+ }
+}
diff --git
a/sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaStaticInitFailingStrLen.java
b/sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaStaticInitFailingStrLen.java
new file mode 100644
index 000000000000..c75d75aa1bbe
--- /dev/null
+++
b/sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaStaticInitFailingStrLen.java
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package test.org.apache.spark.sql.connector.catalog.functions;
+
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.connector.catalog.functions.ScalarFunction;
+import org.apache.spark.sql.connector.catalog.functions.SimpleFunction;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DataTypes;
+
+/**
+ * A string length function whose class fails to initialize.
+ */
+public class JavaStaticInitFailingStrLen implements SimpleFunction,
ScalarFunction<Integer> {
+ static {
+ // javac rejects a static initializer that cannot complete normally; `if`
avoids that.
+ if (true) {
+ throw new IllegalStateException("static initializer failed");
+ }
+ }
+
+ @Override
+ public DataType[] inputTypes() {
+ return new DataType[] { DataTypes.StringType };
+ }
+
+ @Override
+ public DataType resultType() {
+ return DataTypes.IntegerType;
+ }
+
+ @Override
+ public String name() {
+ return "strlen";
+ }
+
+ @Override
+ public String description() {
+ return "strlen(string) -> int";
+ }
+
+ @Override
+ public Integer produceResult(InternalRow input) {
+ return input.getUTF8String(0).numChars();
+ }
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/CreateV2FunctionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/CreateV2FunctionSuite.scala
new file mode 100644
index 000000000000..675bcb6803ca
--- /dev/null
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/CreateV2FunctionSuite.scala
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import java.io.File
+import java.lang.management.ManagementFactory
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.reflect.ClassTag
+
+import
test.org.apache.spark.sql.connector.catalog.functions.{JavaInitTrackingStrLen,
JavaStaticInitFailingStrLen}
+import
test.org.apache.spark.sql.connector.catalog.functions.JavaStrLen.{JavaStrLenNoImpl,
JavaStrLenStaticMagic}
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.catalyst.{FunctionIdentifier, InternalRow}
+import org.apache.spark.sql.connector.catalog.functions.{BoundFunction,
ScalarFunction, SimpleFunction, UnboundFunction}
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DataType, IntegerType, StringType,
StructType}
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.{SparkTestUtils, Utils}
+
+/**
+ * Tests `CREATE [TEMPORARY] FUNCTION name AS 'class'` where the class is a V2
function, i.e. it
+ * implements [[UnboundFunction]].
+ */
+class CreateV2FunctionSuite extends QueryTest with SharedSparkSession {
+
+ private val strLenClass = classOf[V2StrLen].getName
+ private val strLenMagicClass = classOf[V2StrLenMagic].getName
+ private val avgClass = classOf[V2IntegralAverage].getName
+
+ // Runs on executors, which must load the class from the jar too.
+ private val jarStrLenQuery = "SELECT jar_strlen(CAST(id AS STRING)) FROM
range(9, 11)"
+
+ /**
+ * Compiles a V2 string length function named `className` into a jar that is
not on the
+ * classpath, and returns the jar URI. The jar is named after the class, as
Spark rejects
+ * registering the same jar name with a different path. With `magic`, the
function implements
+ * the magic `invoke` method, otherwise `produceResult`. With `requires`,
initializing the class
+ * loads that class through the context classloader.
+ */
+ private def createStrLenJar(
+ className: String,
+ magic: Boolean = true,
+ requires: Option[String] = None): String = {
+ val dir = Utils.createTempDir()
+ val cp = ManagementFactory.getRuntimeMXBean.getClassPath
+ .split(File.pathSeparator).map(p => new File(p).toURI.toURL).toSeq
+ val simpleName = className.split('.').last
+ val method = if (magic) {
+ "public int invoke(UTF8String s) { return s.numChars(); }"
+ } else {
+ "public Integer produceResult(InternalRow r) { return
r.getUTF8String(0).numChars(); }"
+ }
+ // scalastyle:off classforname
+ val staticInit = requires.map { required =>
+ s"""static {
+ | try {
+ | Class.forName("$required", true,
Thread.currentThread().getContextClassLoader());
+ | } catch (ClassNotFoundException e) {
+ | throw new IllegalStateException(e);
+ | }
+ | }""".stripMargin
+ }.getOrElse("")
+ // scalastyle:on classforname
+ val source =
+ s"""package ${className.stripSuffix("." + simpleName)};
+ |
+ |import org.apache.spark.sql.catalyst.InternalRow;
+ |import
org.apache.spark.sql.connector.catalog.functions.ScalarFunction;
+ |import
org.apache.spark.sql.connector.catalog.functions.SimpleFunction;
+ |import org.apache.spark.sql.types.DataType;
+ |import org.apache.spark.sql.types.DataTypes;
+ |import org.apache.spark.unsafe.types.UTF8String;
+ |
+ |public class $simpleName implements SimpleFunction,
ScalarFunction<Integer> {
+ | $staticInit
+ | public DataType[] inputTypes() { return new DataType[] {
DataTypes.StringType }; }
+ | public DataType resultType() { return DataTypes.IntegerType; }
+ | public String name() { return "jar_strlen"; }
+ | public String description() { return "jar_strlen(string) -> int"; }
+ | $method
+ |}
+ |""".stripMargin
+ SparkTestUtils.createJarWithJavaSources(
+ Map(className -> source), new File(dir, s"$simpleName.jar"),
cp).toURI.toString
+ }
+
+ test("temporary scalar function") {
+ withUserDefinedFunction("strlen" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION strlen AS '$strLenClass'")
+ checkAnswer(sql("SELECT strlen('abc')"), Row(3))
+ }
+ }
+
+ test("temporary scalar function with magic method") {
+ withUserDefinedFunction("strlen" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION strlen AS '$strLenMagicClass'")
+ checkAnswer(sql("SELECT strlen('abcd')"), Row(4))
+ }
+ }
+
+ test("persistent scalar function") {
+ withUserDefinedFunction("default.strlen" -> false) {
+ sql(s"CREATE FUNCTION strlen AS '$strLenClass'")
+ checkAnswer(sql("SELECT strlen('abc')"), Row(3))
+ checkAnswer(sql("SELECT default.strlen('abcde')"), Row(5))
+ checkAnswer(sql("SELECT spark_catalog.default.strlen('ab')"), Row(2))
+ }
+ }
+
+ test("CREATE OR REPLACE FUNCTION") {
+ withUserDefinedFunction("default.f" -> false) {
+ sql(s"CREATE FUNCTION f AS '$strLenClass'")
+ checkAnswer(sql("SELECT f('abc')"), Row(3))
+ sql(s"CREATE OR REPLACE FUNCTION f AS '$avgClass'")
+ checkAnswer(sql("SELECT f(id) FROM VALUES (1), (3) AS t(id)"), Row(2))
+ }
+ }
+
+ test("aggregate function") {
+ withUserDefinedFunction("iavg" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION iavg AS '$avgClass'")
+ checkAnswer(sql("SELECT iavg(id) FROM VALUES (1), (2), (3), (5) AS
t(id)"), Row(2))
+ // DISTINCT and FILTER are supported by aggregate functions.
+ checkAnswer(
+ sql("SELECT iavg(DISTINCT id) FROM VALUES (1), (1), (5) AS t(id)"),
Row(3))
+ checkAnswer(
+ sql("SELECT iavg(id) FILTER (WHERE id > 1) FROM VALUES (1), (2), (4)
AS t(id)"), Row(3))
+ // The bound function is chosen by the input type.
+ checkAnswer(sql("SELECT iavg(id) FROM VALUES (1L), (4L) AS t(id)"),
Row(2L))
+ }
+ }
+
+ test("SimpleFunction") {
+ withUserDefinedFunction("strlen" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION strlen AS
'${classOf[V2SimpleStrLen].getName}'")
+ checkAnswer(sql("SELECT strlen('abc')"), Row(3))
+ // Without `bind`, the arguments are implicitly cast to the declared
input types.
+ checkAnswer(sql("SELECT strlen(1234)"), Row(4))
+ val e = intercept[AnalysisException](sql("SELECT strlen('a', 'b')"))
+ assert(e.getCondition == "_LEGACY_ERROR_TEMP_1199")
+ }
+ }
+
+ test("temporary function USING JAR") {
+ val className = "test.v2udf.TempStrLen"
+ assert(!Utils.classIsLoadable(className))
+ val jar = createStrLenJar(className)
+ withUserDefinedFunction("jar_strlen" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION jar_strlen AS '$className' USING JAR
'$jar'")
+ checkAnswer(sql(jarStrLenQuery), Row(1) :: Row(2) :: Nil)
+ }
+ }
+
+ test("temporary function after ADD JAR") {
+ val className = "test.v2udf.AddJarStrLen"
+ assert(!Utils.classIsLoadable(className))
+ sql(s"ADD JAR '${createStrLenJar(className)}'")
+ withUserDefinedFunction("jar_strlen" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION jar_strlen AS '$className'")
+ checkAnswer(sql(jarStrLenQuery), Row(1) :: Row(2) :: Nil)
+ }
+ }
+
+ test("persistent function USING JAR") {
+ val className = "test.v2udf.PersistentStrLen"
+ assert(!Utils.classIsLoadable(className))
+ // Without the magic method, so that the generated code does not refer to
the class.
+ val jar = createStrLenJar(className, magic = false)
+ Utils.withContextClassLoader(Utils.getSparkClassLoader) {
+ withUserDefinedFunction("default.jar_strlen" -> false) {
+ sql(s"CREATE FUNCTION jar_strlen AS '$className' USING JAR '$jar'")
+ def describe(): Seq[String] =
+ sql("DESCRIBE FUNCTION
jar_strlen").collect().map(_.getString(0)).toSeq
+ // The jar is loaded on first invocation, so the description is not
available yet.
+ assert(!describe().exists(_.contains("jar_strlen(string) -> int")))
+
+ checkAnswer(sql(jarStrLenQuery), Row(1) :: Row(2) :: Nil)
+ assert(describe().exists(_.contains("jar_strlen(string) -> int")))
+
+ // Loading the jar switches the context classloader; the function must
still work
+ // after it is rolled back, as spark-shell does.
+ Thread.currentThread().setContextClassLoader(Utils.getSparkClassLoader)
+ checkAnswer(sql(jarStrLenQuery), Row(1) :: Row(2) :: Nil)
+ }
+ }
+ }
+
+ test("magic method from a runtime jar after the context classloader is
reset") {
+ val className = "test.v2udf.ResetLoaderStrLen"
+ val jar = createStrLenJar(className)
+ Utils.withContextClassLoader(Utils.getSparkClassLoader) {
+ withUserDefinedFunction("default.jar_strlen" -> false) {
+ sql(s"CREATE FUNCTION jar_strlen AS '$className' USING JAR '$jar'")
+ checkAnswer(sql(jarStrLenQuery), Row(1) :: Row(2) :: Nil)
+ // Resets the context classloader, as spark-shell does after each
line. Generated code
+ // cannot see the jar then.
+ // TODO(SPARK-60811): the query should succeed once generated code can
see the jar.
+ Thread.currentThread().setContextClassLoader(Utils.getSparkClassLoader)
+ val e = intercept[Exception](sql(jarStrLenQuery).collect())
+ assert(e.getCause.getMessage.contains("Cannot determine simple type
name"))
+ }
+ }
+ }
+
+ test("DESCRIBE FUNCTION reports the V2 function description once it is
loaded") {
+ withUserDefinedFunction("tstrlen" -> true, "default.strlen" -> false) {
+ def describe(name: String): Seq[String] =
+ sql(s"DESCRIBE FUNCTION $name").collect().map(_.getString(0)).toSeq
+
+ sql(s"CREATE TEMPORARY FUNCTION tstrlen AS '$strLenClass'")
+ assert(describe("tstrlen").exists(_.contains("strlen(string) -> int")))
+
+ sql(s"CREATE FUNCTION strlen AS '$strLenClass'")
+ // A persistent function is loaded on first invocation in the session.
+ assert(describe("default.strlen").exists(_.contains(strLenClass)))
+ assert(!describe("default.strlen").exists(_.contains("strlen(string) ->
int")))
+ checkAnswer(sql("SELECT strlen('abc')"), Row(3))
+ assert(describe("default.strlen").exists(_.contains("strlen(string) ->
int")))
+ }
+ }
+
+ test("metadata reads do not construct a persistent function before its first
invocation") {
+ V2MetadataTrackingStrLen.constructions.set(0)
+ val className = classOf[V2MetadataTrackingStrLen].getName
+ withUserDefinedFunction("default.metadata_len" -> false) {
+ sql(s"CREATE FUNCTION metadata_len AS '$className'")
+ assert(spark.catalog.functionExists("metadata_len"))
+ assert(spark.catalog.functionExists("default", "metadata_len"))
+ assert(spark.catalog.listFunctions().collect().exists(_.name ==
"metadata_len"))
+ assert(spark.catalog.getFunction("metadata_len").className == className)
+ sql("DESCRIBE FUNCTION EXTENDED metadata_len").collect()
+ assert(V2MetadataTrackingStrLen.constructions.get() == 0)
+ checkAnswer(sql("SELECT metadata_len('abc')"), Row(3))
+ assert(V2MetadataTrackingStrLen.constructions.get() == 1)
+ assert(spark.catalog.getFunction("metadata_len").className == className)
+ assert(V2MetadataTrackingStrLen.constructions.get() == 1)
+ }
+ }
+
+ test("metadata reads do not initialize the class of a persistent function") {
+ val className = "test.v2udf.RequiresLaterJarStrLen"
+ val requiredClassName = "test.v2udf.LaterJarStrLen"
+ // Initializing the class fails until the jar of the class it requires is
added. A failed
+ // initialization would make the class unusable for the rest of the JVM's
life.
+ sql(s"ADD JAR '${createStrLenJar(className, requires =
Some(requiredClassName))}'")
+ withUserDefinedFunction("default.metadata_len" -> false) {
+ sql(s"CREATE FUNCTION metadata_len AS '$className'")
+ assert(spark.catalog.listFunctions().collect().exists(_.name ==
"metadata_len"))
+ assert(spark.catalog.getFunction("metadata_len").description == "N/A.")
+ sql("DESCRIBE FUNCTION EXTENDED metadata_len").collect()
+
+ sql(s"ADD JAR '${createStrLenJar(requiredClassName)}'")
+ checkAnswer(sql("SELECT metadata_len('abc')"), Row(3))
+ assert(spark.catalog.getFunction("metadata_len").description ==
"jar_strlen(string) -> int")
+ }
+ }
+
+ gridTest("class whose description throws")(
+ Seq(classOf[V2FunctionThrowingDescription].getName,
+ classOf[V2FunctionLinkageErrorDescription].getName)) { className =>
+ withUserDefinedFunction("tstrlen" -> true, "default.pstrlen" -> false) {
+ sql(s"CREATE TEMPORARY FUNCTION tstrlen AS '$className'")
+ sql(s"CREATE FUNCTION pstrlen AS '$className'")
+
+ def checkUnavailableDescription(name: String): Unit = {
+ val output = sql(s"DESCRIBE FUNCTION
$name").collect().map(_.getString(0))
+ assert(output.contains(s"Class: $className"))
+ assert(output.contains("Usage: N/A."))
+ assert(spark.catalog.getFunction(name).className == className)
+ val info =
spark.sessionState.catalog.lookupFunctionInfo(FunctionIdentifier(name))
+ assert(info.getSource == "v2_function")
+ }
+
+ checkUnavailableDescription("tstrlen")
+ checkAnswer(sql("SELECT tstrlen(CAST(id AS STRING)) FROM range(9, 11)"),
+ Row(1) :: Row(2) :: Nil)
+ checkAnswer(sql("SELECT pstrlen(CAST(id AS STRING)) FROM range(9, 11)"),
+ Row(1) :: Row(2) :: Nil)
+ checkUnavailableDescription("pstrlen")
+
+ sql("REFRESH FUNCTION pstrlen")
+ checkAnswer(sql("SELECT pstrlen('abc')"), Row(3))
+ checkUnavailableDescription("pstrlen")
+ }
+ }
+
+ test("DROP FUNCTION") {
+ sql(s"CREATE FUNCTION strlen AS '$strLenClass'")
+ checkAnswer(sql("SELECT strlen('abc')"), Row(3))
+ sql("DROP FUNCTION strlen")
+ intercept[AnalysisException](sql("SELECT strlen('abc')"))
+ }
+
+ test("unsupported clauses and column names use the function name") {
+ withUserDefinedFunction("strlen" -> true, "iavg" -> true,
"default.pstrlen" -> false) {
+ sql(s"CREATE TEMPORARY FUNCTION strlen AS '$strLenClass'")
+ sql(s"CREATE TEMPORARY FUNCTION iavg AS '$avgClass'")
+ sql(s"CREATE FUNCTION pstrlen AS '$strLenClass'")
+ assert(sql("SELECT strlen('a'), pstrlen('a'), iavg(id) FROM
range(3)").columns.toSeq ==
+ Seq("strlen(a)", "spark_catalog.default.pstrlen(a)", "iavg(id)"))
+
+ def checkUnsupported(call: String, name: String, syntax: String): Unit =
{
+ checkError(
+ exception = intercept[AnalysisException](sql(s"SELECT $call FROM
range(3)")),
+ condition = "INVALID_SQL_SYNTAX.FUNCTION_WITH_UNSUPPORTED_SYNTAX",
+ parameters = Map("prettyName" -> name, "syntax" -> syntax),
+ context = ExpectedContext(call, 7, 6 + call.length))
+ }
+ checkUnsupported("strlen(DISTINCT 'a')", "`strlen`", "DISTINCT")
+ checkUnsupported("pstrlen(DISTINCT 'a')",
"`spark_catalog`.`default`.`pstrlen`", "DISTINCT")
+ checkUnsupported("strlen('a') FILTER (WHERE true)", "`strlen`", "FILTER
CLAUSE")
+ checkUnsupported("iavg(id) IGNORE NULLS", "`iavg`", "IGNORE NULLS")
+ checkUnsupported(
+ "iavg(id) WITHIN GROUP (ORDER BY id)", "`iavg`", "WITHIN GROUP (ORDER
BY ...)")
+ }
+ }
+
+ test("column names of magic methods use the function name") {
+ val simpleMagicClass = classOf[V2SimpleStrLenMagic].getName
+ withUserDefinedFunction(
+ "strlen" -> true, "sstrlen" -> true, "simple_strlen" -> true,
"default.pstrlen" -> false) {
+ sql(s"CREATE TEMPORARY FUNCTION strlen AS '$strLenMagicClass'")
+ sql(s"CREATE TEMPORARY FUNCTION sstrlen AS
'${classOf[V2StrLenStaticMagic].getName}'")
+ sql(s"CREATE TEMPORARY FUNCTION simple_strlen AS '$simpleMagicClass'")
+ sql(s"CREATE FUNCTION pstrlen AS '$strLenMagicClass'")
+ val df = sql("SELECT strlen('a'), sstrlen('a'), simple_strlen('a'),
pstrlen('a')")
+ assert(df.columns.toSeq ==
+ Seq("strlen(a)", "sstrlen(a)", "simple_strlen(a)",
"spark_catalog.default.pstrlen(a)"))
+
+ // The column name does not depend on the function instance.
+ withTempView("v") {
+ sql("CREATE TEMPORARY VIEW v AS SELECT simple_strlen(s) FROM VALUES
('abc') t(s)")
+ sql(s"CREATE OR REPLACE TEMPORARY FUNCTION simple_strlen AS
'$simpleMagicClass'")
+ checkAnswer(spark.table("v"), Row(3))
+ }
+ }
+ }
+
+ gridTest("bind-time errors use the registered function name")(for {
+ isTemp <- Seq(true, false)
+ funcName <- Seq("my_len", "my.len", "my`len")
+ } yield (isTemp, funcName)) { case (isTemp, funcName) =>
+ val ident = FunctionIdentifier(funcName, if (isTemp) None else
Some("default"))
+ val name = ident.quotedString
+ val registeredIdent = if (isTemp) ident else ident.copy(catalog =
Some("spark_catalog"))
+ val registeredName = registeredIdent.unquotedString
+ val quotedName = registeredIdent.quotedString
+ val create = if (isTemp) "CREATE TEMPORARY FUNCTION" else "CREATE FUNCTION"
+
+ def checkBindingError(
+ className: String,
+ args: String,
+ condition: String,
+ parameters: Map[String, String]): Unit = {
+ withUserDefinedFunction(name -> isTemp) {
+ sql(s"$create $name AS '$className'")
+ val call = s"$name($args)"
+ checkError(
+ exception = intercept[AnalysisException](sql(s"SELECT $call")),
+ condition = condition,
+ parameters = parameters,
+ context = ExpectedContext(call, 7, 6 + call.length))
+ }
+ }
+
+ checkBindingError(strLenClass, "1", "_LEGACY_ERROR_TEMP_1198",
+ Map("unbound" -> registeredName, "arguments" -> "int", "unsupported" ->
"Expect StringType"))
+ checkBindingError(classOf[V2SimpleStrLen].getName, "'a', 'b'",
"_LEGACY_ERROR_TEMP_1199",
+ Map("bound" -> registeredName, "argsLen" -> "2", "inputTypesLen" -> "1"))
+ checkBindingError(classOf[V2StrLenBadBound].getName, "'abc'",
"INVALID_UDF_IMPLEMENTATION",
+ Map("funcName" -> quotedName))
+ checkBindingError(
+ classOf[V2StrLenNoImpl].getName, "'abc'",
"SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED",
+ Map("scalarFunc" -> quotedName))
+ }
+
+ test("input type that the function cannot bind") {
+ withUserDefinedFunction("strlen" -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION strlen AS '$strLenClass'")
+ val e = intercept[AnalysisException](sql("SELECT strlen(1)"))
+ assert(e.getMessage.contains("Expect StringType"))
+ val e2 = intercept[AnalysisException](sql("SELECT strlen('a', 'b')"))
+ assert(e2.getMessage.contains("Expect exactly one argument"))
+ }
+ }
+
+ /**
+ * Checks that a broken persistent function `f` can be created, listed and
described like a
+ * Hive UDF, and only fails when invoked.
+ */
+ private def checkFailsOnlyWhenInvoked[T <: Throwable : ClassTag](createSql:
String)(
+ check: T => Unit): Unit = {
+ withUserDefinedFunction("default.f" -> false) {
+ sql(createSql)
+ assert(sql("SHOW USER FUNCTIONS").collect().map(_.getString(0)).contains(
+ "spark_catalog.default.f"))
+ assert(spark.catalog.listFunctions().collect().exists(_.name == "f"))
+ assert(spark.catalog.getFunction("f").className ==
createSql.split("'")(1))
+ sql("DESCRIBE FUNCTION EXTENDED f").collect()
+ check(intercept[T](sql("SELECT f('abc')").collect()))
+ }
+ }
+
+ /** Checks that a class that cannot be instantiated fails as
`CANNOT_INSTANTIATE_...`. */
+ private def checkCannotInstantiate(className: String, msg: String): Unit = {
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"CREATE TEMPORARY FUNCTION f AS '$className'")
+ },
+ condition = "CANNOT_INSTANTIATE_FUNCTION_CLASS",
+ parameters = Map("className" -> className, "functionName" -> "`f`",
"msg" -> msg))
+
+ checkFailsOnlyWhenInvoked[AnalysisException](s"CREATE FUNCTION f AS
'$className'") { e =>
+ checkError(
+ exception = e,
+ condition = "CANNOT_INSTANTIATE_FUNCTION_CLASS",
+ parameters = Map(
+ "className" -> className,
+ "functionName" -> "`spark_catalog`.`default`.`f`",
+ "msg" -> msg),
+ context = ExpectedContext("f('abc')", 7, 14))
+ }
+ }
+
+ test("class without a public no-arg constructor") {
+ val className = classOf[V2FunctionWithoutNoArgConstructor].getName
+ checkCannotInstantiate(className, s"$className.<init>()")
+ }
+
+ test("class whose constructor throws") {
+ checkCannotInstantiate(classOf[V2FunctionThrowingConstructor].getName,
"constructor failed")
+ }
+
+ test("class whose constructor throws a linkage error") {
+ checkCannotInstantiate(
+ classOf[V2FunctionLinkageErrorConstructor].getName, "constructor
dependency")
+ }
+
+ gridTest("fatal constructor failures propagate")(
+ Seq(
+ (classOf[V2FunctionOutOfMemoryConstructor].getName,
classOf[OutOfMemoryError]),
+ (classOf[V2FunctionStackOverflowConstructor].getName,
classOf[StackOverflowError]),
+ (classOf[V2FunctionInterruptedConstructor].getName,
classOf[InterruptedException]))) {
+ case (className, errorClass) =>
+ def checkFatal(operation: => Any): Unit = {
+ val e = intercept[Throwable](operation)
+ assert(e.getClass == errorClass)
+ assert(e.getMessage == "constructor failed")
+ }
+
+ checkFatal(sql(s"CREATE TEMPORARY FUNCTION f AS '$className'"))
+ withUserDefinedFunction("default.f" -> false) {
+ sql(s"CREATE FUNCTION f AS '$className'")
+ sql("DESCRIBE FUNCTION f").collect()
+ spark.catalog.getFunction("f")
+ spark.catalog.listFunctions().collect()
+ checkFatal(sql("SELECT f('abc')").collect())
+ checkFatal(sql("REFRESH FUNCTION f"))
+ }
+ }
+
+ test("class that fails to initialize") {
+ val className = classOf[JavaStaticInitFailingStrLen].getName
+ checkFailsOnlyWhenInvoked[LinkageError](s"CREATE FUNCTION f AS
'$className'") { _ => }
+ }
+
+ test("USING resources are loaded before the class is initialized") {
+ val className = classOf[JavaInitTrackingStrLen].getName
+ val initialized = "spark.test.JavaInitTrackingStrLen.initialized"
+ // Initializing the class requires a class in this jar, so it fails if
done before the jar
+ // is loaded.
+ val jar = createStrLenJar("test.v2udf.UsingResourceStrLen")
+ withUserDefinedFunction("default.f" -> false) {
+ sql(s"CREATE FUNCTION f AS '$className' USING JAR '$jar'")
+ assert(spark.catalog.functionExists("f"))
+ spark.catalog.listFunctions().collect()
+ spark.catalog.getFunction("f")
+ val describe = sql("DESCRIBE FUNCTION f").collect().map(_.getString(0))
+ assert(!describe.exists(_.contains("strlen(string) -> int")))
+ assert(System.getProperty(initialized) == null)
+
+ checkAnswer(sql("SELECT f('abc')"), Row(3))
+ assert(System.getProperty(initialized) == "true")
+ assert(sql("DESCRIBE FUNCTION f").collect().map(_.getString(0))
+ .exists(_.contains("strlen(string) -> int")))
+ }
+ }
+
+ test("class that is not on the classpath") {
+ checkFailsOnlyWhenInvoked[AnalysisException]("CREATE FUNCTION f AS
'not.a.Class'") { e =>
+ checkError(
+ exception = e,
+ condition = "CANNOT_LOAD_FUNCTION_CLASS",
+ parameters = Map(
+ "className" -> "not.a.Class",
+ "functionName" -> "`spark_catalog`.`default`.`f`"),
+ context = ExpectedContext("f('abc')", 7, 14))
+ }
+ }
+
+ test("jar that is not accessible") {
+ val jar = new File(Utils.createTempDir(), "missing.jar").toURI.toString
+ checkFailsOnlyWhenInvoked[AnalysisException](
+ s"CREATE FUNCTION f AS 'test.v2udf.MissingJarStrLen' USING JAR
'$jar'") { e =>
+ assert(e.getCondition == "CANNOT_LOAD_FUNCTION_CLASS", e.getMessage)
+ }
+ }
+}
+
+class V2StrLen extends UnboundFunction {
+ override def name(): String = "strlen"
+
+ override def bind(inputType: StructType): BoundFunction = {
+ if (inputType.fields.length != 1) {
+ throw new UnsupportedOperationException("Expect exactly one argument")
+ }
+ inputType.fields(0).dataType match {
+ case _: StringType => StrLenDefault
+ case _ => throw new UnsupportedOperationException("Expect StringType")
+ }
+ }
+
+ override def description(): String = "strlen: strlen(string) -> int"
+}
+
+object V2MetadataTrackingStrLen {
+ val constructions = new AtomicInteger
+}
+
+class V2MetadataTrackingStrLen extends V2StrLen {
+ V2MetadataTrackingStrLen.constructions.incrementAndGet()
+}
+
+class V2StrLenMagic extends V2StrLen {
+ override def bind(inputType: StructType): BoundFunction = {
+ super.bind(inputType)
+ StrLenMagic
+ }
+}
+
+class V2StrLenBadBound extends V2StrLen {
+ override def bind(inputType: StructType): BoundFunction = BadBoundFunction
+}
+
+class V2StrLenNoImpl extends V2StrLen {
+ override def bind(inputType: StructType): BoundFunction = new
JavaStrLenNoImpl
+}
+
+class V2SimpleStrLen extends SimpleFunction with ScalarFunction[Int] {
+ override def inputTypes(): Array[DataType] = Array(StringType)
+ override def resultType(): DataType = IntegerType
+ override def name(): String = "strlen"
+ override def description(): String = "strlen: strlen(string) -> int"
+ override def produceResult(input: InternalRow): Int =
input.getUTF8String(0).numChars()
+}
+
+class V2StrLenStaticMagic extends V2StrLen {
+ override def bind(inputType: StructType): BoundFunction = {
+ super.bind(inputType)
+ new JavaStrLenStaticMagic
+ }
+}
+
+class V2SimpleStrLenMagic extends SimpleFunction with ScalarFunction[Int] {
+ override def inputTypes(): Array[DataType] = Array(StringType)
+ override def resultType(): DataType = IntegerType
+ override def name(): String = "strlen"
+ override def description(): String = "strlen: strlen(string) -> int"
+ def invoke(s: UTF8String): Int = s.numChars()
+}
+
+class V2IntegralAverage extends UnboundFunction {
+ override def name(): String = IntegralAverage.name()
+ override def bind(inputType: StructType): BoundFunction =
IntegralAverage.bind(inputType)
+ override def description(): String = IntegralAverage.description()
+}
+
+class V2FunctionThrowingDescription extends V2StrLen {
+ override def description(): String = throw new
UnsupportedOperationException("description failed")
+}
+
+class V2FunctionLinkageErrorDescription extends V2StrLen {
+ // scalastyle:off throwerror
+ override def description(): String = throw new
NoClassDefFoundError("description dependency")
+ // scalastyle:on throwerror
+}
+
+class V2FunctionThrowingConstructor extends V2StrLen {
+ throw new IllegalStateException("constructor failed")
+}
+
+class V2FunctionLinkageErrorConstructor extends V2StrLen {
+ // scalastyle:off throwerror
+ throw new NoClassDefFoundError("constructor dependency")
+ // scalastyle:on throwerror
+}
+
+class V2FunctionOutOfMemoryConstructor extends V2StrLen {
+ // scalastyle:off throwerror
+ throw new OutOfMemoryError("constructor failed")
+ // scalastyle:on throwerror
+}
+
+class V2FunctionStackOverflowConstructor extends V2StrLen {
+ // scalastyle:off throwerror
+ throw new StackOverflowError("constructor failed")
+ // scalastyle:on throwerror
+}
+
+class V2FunctionInterruptedConstructor extends V2StrLen {
+ throw new InterruptedException("constructor failed")
+}
+
+class V2FunctionWithoutNoArgConstructor(name: String) extends UnboundFunction {
+ override def name(): String = name
+ override def bind(inputType: StructType): BoundFunction = StrLenDefault
+ override def description(): String = name
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
index 7b5402119b40..49d242ad4511 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
@@ -83,7 +83,7 @@ class ExpressionInfoSuite extends SharedSparkSession {
val validSources = Seq(
"built-in", "hive", "python_udf", "scala_udf", "java_udf",
"python_udtf", "internal",
- "sql_udf")
+ "sql_udf", "v2_function")
validSources.foreach { source =>
val info = new ExpressionInfo(
"testClass", null, "testName", null, "", "", "", "", "", "", source)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
index 89edb53048a3..5b5474512f29 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -42,9 +42,10 @@ import org.apache.hadoop.io.{LongWritable, Writable}
import org.apache.spark.{SparkException, SparkFiles, SparkRuntimeException,
TestUtils}
import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
BindReferences, CodegenObjectFactoryMode, Literal}
+import org.apache.spark.sql.catalyst.expressions.{ApplyFunctionExpression,
AttributeReference, BindReferences, CodegenObjectFactoryMode, Literal}
import org.apache.spark.sql.catalyst.plans.logical.{Filter, Project}
import org.apache.spark.sql.catalyst.util.{DateTimeUtils, GenericArrayData}
+import org.apache.spark.sql.connector.catalog.functions.{ScalarFunction,
SimpleFunction}
import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.functions.{call_function, max}
import org.apache.spark.sql.hive.{HiveGenericUDF, HiveGenericUDTF}
@@ -1250,6 +1251,38 @@ class HiveUDFSuite extends QueryTest with
TestHiveSingleton {
}
hiveContext.reset()
}
+
+ test("class implementing both GenericUDF and a V2 function runs as a V2
function") {
+ withUserDefinedFunction("dual_strlen" -> false) {
+ sql(s"CREATE FUNCTION dual_strlen AS
'${classOf[GenericUDFAndV2StrLen].getName}'")
+ val df = sql("SELECT dual_strlen('abc')")
+ val analyzed = df.queryExecution.analyzed
+
assert(!analyzed.exists(_.expressions.exists(_.exists(_.isInstanceOf[HiveGenericUDF]))))
+ assert(analyzed.exists(_.expressions.exists(_.exists(
+ _.isInstanceOf[ApplyFunctionExpression]))))
+ checkAnswer(df, Row(3))
+ }
+ }
+}
+
+/**
+ * A function that runs in Hive as a [[GenericUDF]] and in Spark as a V2
function.
+ */
+class GenericUDFAndV2StrLen extends GenericUDF with SimpleFunction with
ScalarFunction[Int] {
+ override def initialize(arguments: Array[ObjectInspector]): ObjectInspector =
+ PrimitiveObjectInspectorFactory.javaIntObjectInspector
+
+ override def evaluate(arguments: Array[DeferredObject]): AnyRef =
+ Int.box(arguments(0).get().toString.length)
+
+ override def getDisplayString(children: Array[String]): String =
+ s"dual_strlen(${children.mkString(", ")})"
+
+ override def inputTypes(): Array[DataType] = Array(StringType)
+ override def resultType(): DataType = IntegerType
+ override def name(): String = "dual_strlen"
+ override def description(): String = "dual_strlen(string) -> int"
+ override def produceResult(input: InternalRow): Int =
input.getUTF8String(0).numChars()
}
class TestPair(x: Int, y: Int) extends Writable with Serializable {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]