[
https://issues.apache.org/jira/browse/HIVE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15517776#comment-15517776
]
Jason Dere commented on HIVE-14819:
-----------------------------------
I see another issue related to being able to determine if a function is a
permanent UDF:
When Hive attempts to register all of the existing permanent UDFs during Hive
initialization, it does not download the JARs and add them to the classloader.
However as you can see from the stack trace, it does try to get the UDF class
in order to add it to this persistent UDFs list. This fails since Hive has not
loaded the resources. As far as I can tell, this is going to fail for any
permanent UDFs which require JARs which are not in Hive's classpath at
initialization time.
While this does not prevent permanent functions from working in most cases
(dynamic lookup works and it's added to the session registry), this does cause
problems with being able to tell if this is a permanent function because the
UDF's class is never added to the system registry's list of persistent
functions. Thus FunctionRegistry.isPermanentFunction() does not work properly.
{noformat}
016-09-23T00:32:47,869 WARN [main]: metadata.Hive
(Hive.java:reloadFunctions(247)) - Failed to register persistent function
sleep2:org.apache.hive.udf.generic.GenericUDFSleep. Ignore and continue.
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hive.udf.generic.GenericUDFSleep
at
org.apache.hadoop.hive.ql.exec.Registry.getPermanentUdfClass(Registry.java:483)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.exec.Registry.addFunction(Registry.java:466)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:239)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1511)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:243)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:224)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:369)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:313)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:293)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:269)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
~[hive-cli-2.1.0.2.5.1.0-46.jar:2.1.0.2.5.1.0-46]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
~[hive-cli-2.1.0.2.5.1.0-46.jar:2.1.0.2.5.1.0-46]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_91]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_91]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
~[hadoop-common-2.7.3.2.5.1.0-46.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
~[hadoop-common-2.7.3.2.5.1.0-46.jar:?]
Caused by: java.lang.ClassNotFoundException:
org.apache.hive.udf.generic.GenericUDFSleep
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
~[?:1.8.0_91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_91]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_91]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_91]
at
org.apache.hadoop.hive.ql.exec.Registry.getPermanentUdfClass(Registry.java:481)
~[hive-exec-2.1.0.jar:2.1.0]
... 19 more
{noformat}
> FunctionInfo for permanent functions shows TEMPORARY FunctionType
> -----------------------------------------------------------------
>
> Key: HIVE-14819
> URL: https://issues.apache.org/jira/browse/HIVE-14819
> Project: Hive
> Issue Type: Bug
> Components: UDF
> Affects Versions: 2.1.0
> Reporter: Jason Dere
> Assignee: Jason Dere
> Attachments: HIVE-14819.1.patch
>
>
> The FunctionInfo has a FunctionType field which describes if the function is
> a builtin/persistent/temporary function. But for permanent functions, the
> FunctionInfo being returned by the FunctionRegistry is showing the type to be
> TEMPORARY.
> This affects things which may be depending on function type, for example
> LlapDecider, which will allow builtin/persistent UDFs to be used in LLAP but
> not temporary functions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
