[
https://issues.apache.org/jira/browse/IMPALA-11342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606899#comment-17606899
]
ASF subversion and git services commented on IMPALA-11342:
----------------------------------------------------------
Commit 7dce6bc8f2051aba533073f5beca99867e730a09 in impala's branch
refs/heads/branch-4.1.1 from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7dce6bc8f ]
IMPALA-11342: Fix class loading in Hive UDFs' constructors
Loading new classes from the same jar in the constructor of UDFs
did not work in the catalog because the URLClassLoader was closed
too early. Extended the lifecycle of the class loader a bit to
let the catalog finish all initialisation.
Note that the instantiation of legacy Hive UDFs doesn't seem
necessary in the catalog, we can get all relevant info from
the class. Generic UDFs do need to be instantiated to be able
to call initialize().
Testing:
- added new classes to load in test UDFs and loaded these
in constructor / initialize()
- ran the Hive UDF ee tests
Merge conflicts in branch-4.1:
- HiveJavaFunctionFactoryImpl.java ignores the case for GenericUDF
- Ignores changes in GenericImportsNearbyClassesUdf.java
Change-Id: If16e38b8fc3b2577a5d32104ea9e6948b9562e24
Reviewed-on: http://gerrit.cloudera.org:8080/18611
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/19009
Reviewed-by: Csaba Ringhofer <[email protected]>
> Hive UDFs are unable to use classes from the same Jar in catalogd
> -----------------------------------------------------------------
>
> Key: IMPALA-11342
> URL: https://issues.apache.org/jira/browse/IMPALA-11342
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 4.1.0
> Reporter: Csaba Ringhofer
> Assignee: Csaba Ringhofer
> Priority: Major
> Fix For: Impala 4.2.0
>
>
> While it is a known limitations that Hive UDFs in Impala cannot load
> dependencies from classpath, they should be able to load classes from the
> same jar (and thus creating an shaded jar with the dependencies should be a
> valid workaround). Meanwhile I got an error when the parent class of the UDF
> was a class defined in another package in the jar.
> The exception occurred when catalogd tried to instantiate the UDF class:
> {code}
> I0608 14:50:21.520296 22464 jni-util.cc:286] java.lang.NoClassDefFoundError:
> com/esri/core/geometry/Point
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> at java.lang.Class.getConstructor0(Class.java:3075)
> at java.lang.Class.getConstructor(Class.java:1825)
> at
> org.apache.impala.hive.executor.HiveLegacyJavaFunction.instantiateUDFInstance(HiveLegacyJavaFunction.java:115)
> at
> org.apache.impala.hive.executor.HiveLegacyJavaFunction.<init>(HiveLegacyJavaFunction.java:74)
> at
> org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:50)
> at
> org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:67)
> at
> org.apache.impala.service.CatalogOpExecutor.createFunction(CatalogOpExecutor.java:2117)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:453)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:245)
> Caused by: java.lang.ClassNotFoundException: com.esri.core.geometry.Point
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 11 more
> {code}
> The issue seems to be cause by closing the UrlClassLoader too early, after
> loading the class but before doing the actual instantiation:
> https://github.com/apache/impala/blob/13bbff4e4e5fc5d459cc6f7a5512f84ceba897cd/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfLoader.java#L71
> Note that this is an issue only in the catalogd, not the executors, as the
> class loader is kept open longer in that case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
