Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/9664#discussion_r44873198
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -454,7 +454,7 @@ class HiveContext private[hive](
// Note that HiveUDFs will be overridden by functions registered in this
context.
@transient
override protected[sql] lazy val functionRegistry: FunctionRegistry =
- new HiveFunctionRegistry(FunctionRegistry.builtin.copy()) {
+ new HiveFunctionRegistry(FunctionRegistry.builtin.copy(), this) {
--- End diff --
Thanks for pointing this out. At first I didn't notice this part either.
Just reading the code, I'd assume that this already fixes the issue. But it
wasn't the case.
After some investigation, I'm quite puzzled by the behavior here. Without
this PR, we can add a jar, create a UDTF from the jar, and apply this UDTF in
SQL queries successfully. However, `DESCRIBE FUNCTION` still returns "Function:
<function-name> is not found". I tried single-step debugging `DescribeFunction`
and noticed that the `sqlContext.functionRegistry.lookupFunction` call goes
directly to `HiveFunctionRegistry.lookupFunction` without calling the overriden
version defined in this anonymous class.
Anyway, now we can remove this anonymous class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
