pan3793 commented on code in PR #59697:
URL: https://github.com/apache/spark/pull/59697#discussion_r4442451407
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala:
##########
@@ -2167,12 +2228,58 @@ class SessionCatalog(
if (funcDefinition.isUserDefinedFunction) {
UserDefinedFunction.fromCatalogFunction(funcDefinition,
parser).toExpressionInfo
} else {
- makeExprInfoForHiveFunction(funcDefinition)
+ makeExprInfoForClassFunction(funcDefinition)
}
}
registry.registerFunction(identToRegister, resolvedInfo, functionBuilder)
}
+ /**
+ * Builds the [[ExpressionInfo]] of a class-based function (`CREATE FUNCTION
... AS 'class'`).
+ * A V2 function reports its own description, which requires instantiating
it. Otherwise, or if
+ * the class cannot be loaded or instantiated, use the class-name-only info,
so that like a Hive
+ * UDF, a broken function only fails when invoked. The class of a function
with `USING`
+ * resources is not touched, as initializing it may require those resources,
which are only
+ * loaded on first invocation.
+ */
+ private def makeExprInfoForClassFunction(func: CatalogFunction):
ExpressionInfo = {
+ if (func.resources.nonEmpty) {
+ return makeExprInfoForHiveFunction(func)
+ }
+ try {
+ val clazz = Utils.classForName(func.className, initialize = false)
+ if (isV2FunctionClass(clazz)) {
+ makeExprInfoForV2Function(func, newV2FunctionInstance(clazz,
func.identifier))
Review Comment:
@LuciferYang Done in 169f61b8b85. Metadata reads now use the class-name-only
info for a persistent class function until it is invoked in the session, so
they no longer initialize the class; the V2 description shows after the first
invocation. The new test `metadata reads do not initialize the class of a
persistent function` covers your scenario and fails with `NoClassDefFoundError`
without the change. The affected tests and the PR description are updated.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala:
##########
@@ -2073,17 +2077,69 @@ class SessionCatalog(
}
/**
- * Constructs a [[FunctionBuilder]] based on the provided function metadata.
+ * Constructs a [[FunctionBuilder]] and the [[ExpressionInfo]] based on the
provided function
+ * metadata.
*/
- private def makeFunctionBuilder(func: CatalogFunction): FunctionBuilder = {
+ private def makeFunctionBuilderAndInfo(
+ func: CatalogFunction): (FunctionBuilder, ExpressionInfo) = {
+ val clazz = loadFunctionClass(func)
+ val name = func.identifier.unquotedString
+ if (isV2FunctionClass(clazz)) {
+ // A V2 function: bind it to the arguments at each call site. An
aggregate function is
+ // returned as a bare `V2Aggregator`, which the caller wraps into an
`AggregateExpression`
+ // carrying DISTINCT / FILTER. This check goes first so that a class
also implementing a
+ // Hive UDF interface, for other engines sharing the metastore, runs as
a V2 function in
+ // Spark. The alias makes errors and column names use the function name,
like the Hive UDF
+ // wrappers do.
+ val unbound = newV2FunctionInstance(clazz, func.identifier)
+ val builder = (input: Seq[Expression]) => {
+ val expr = V2ExpressionUtils.bindFunction(
+ unbound, input, displayName = Some(func.identifier))
+ expr.setTagValue(FunctionRegistry.FUNC_ALIAS, name)
Review Comment:
@LuciferYang Done in 169f61b8b85. An `Invoke` / `StaticInvoke` carrying
`FUNC_ALIAS` now gets its column name from the function name. The new test
`column names of magic methods use the function name` covers instance and
static magic methods, and the view after `CREATE OR REPLACE TEMPORARY FUNCTION`.
##########
docs/sql-ref-syntax-ddl-create-function.md:
##########
@@ -80,13 +80,22 @@ CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [ IF NOT
EXISTS ]
* **class_name**
Specifies the name of the class that provides the implementation for
function to be created.
- The implementing class should extend one of the base classes as follows:
+ The implementing class should have a public no-arg constructor, and extend
one of the base
+ classes as follows:
+ * Should implement `UnboundFunction` in
`org.apache.spark.sql.connector.catalog.functions`
+ package.
* Should extend `UDF` or `UDAF` in `org.apache.hadoop.hive.ql.exec`
package.
* Should extend `AbstractGenericUDAFResolver`, `GenericUDF`, or
`GenericUDTF` in `org.apache.hadoop.hive.ql.udf.generic` package.
* Should extend `UserDefinedAggregateFunction` in
`org.apache.spark.sql.expressions` package.
+ **Note:** If an `UnboundFunction` binds to a `ScalarFunction` that
implements the magic method
+ `invoke`, and its jar is added at runtime (`ADD JAR`, or `USING JAR`),
queries calling it may
+ fail to compile generated code when run on a thread other than the one
that added the jar,
Review Comment:
@LuciferYang Reworded the note in 169f61b8b85: it covers the whole-stage
codegen fallback and the failing case, without the thread wording. Opened
[SPARK-60811](https://issues.apache.org/jira/browse/SPARK-60811), linked from
the PR description rather than the user doc. Added the `magic = true` test with
a TODO in 8bf21ed06dc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
