Csaba Ringhofer created IMPALA-11342:
----------------------------------------
Summary: Hive UDFs are unable to use classes from the same Jar in
catalogd
Key: IMPALA-11342
URL: https://issues.apache.org/jira/browse/IMPALA-11342
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Csaba Ringhofer
While it is a known limitations that Hive UDFs in Impala cannot load
dependencies from classpath, they should be able to load classes from the same
jar (and thus creating an shaded jar with the dependencies should be a valid
workaround). Meanwhile I got an error when the parent class of the UDF was a
class defined in another package in the jar.
The exception occurred when catalogd tried to instantiate the UDF class:
{code}
I0608 14:50:21.520296 22464 jni-util.cc:286] java.lang.NoClassDefFoundError:
com/esri/core/geometry/Point
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at
org.apache.impala.hive.executor.HiveLegacyJavaFunction.instantiateUDFInstance(HiveLegacyJavaFunction.java:115)
at
org.apache.impala.hive.executor.HiveLegacyJavaFunction.<init>(HiveLegacyJavaFunction.java:74)
at
org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:50)
at
org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:67)
at
org.apache.impala.service.CatalogOpExecutor.createFunction(CatalogOpExecutor.java:2117)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:453)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:245)
Caused by: java.lang.ClassNotFoundException: com.esri.core.geometry.Point
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
{code}
The issue seems to be cause by closing the UrlClassLoader too early, after
loading the class but before doing the actual instantiation:
https://github.com/apache/impala/blob/13bbff4e4e5fc5d459cc6f7a5512f84ceba897cd/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfLoader.java#L71
Note that this is an issue only in the catalogd, not the executors, as the
class loader is kept open longer in that case.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
