Hello Quanlong Huang, Daniel Becker, Peter Rozsa, Steve Carlin, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19177
to look at the new patch set (#7).
Change subject: WIP IMPALA-11718: Improve support for Hive Generic UDFs
......................................................................
WIP IMPALA-11718: Improve support for Hive Generic UDFs
Hive Generic UDFs provides generic functionality using function
initialize(), which get the descriptors of input types, a.k.a.
ObjectInspectors. Based on the input types it initializes its
state and returns an ObjectInspector that describes the return type.
IMPALA-11162 has provided support for loading and running Hive generic
UDFs, but it did not add support for handling them in an actually
generic way, leading to several limitations. It expected both argument
and return types to be set during creating and used initialize() only
to validate that this signiture is supported.
This patch improves support in the following ways:
- Hive Generic UDFs can be created without specifying argument and
return types. In this case initialize() will be called during
analyses of the query with the expected argument types.
- When created this way, the UDFs can be persisted in HMS.
SHOW FUNCTIONS returns generic UDFs expecting and returning a
"GENERIC" type. Hive's SHOW FUNCTION does not return argument / return
types, so its behavoir could not be copied.
When resolving the function during analyses, the "old" type of functions
take precedence to avoid breaking existing workloads.
Testing:
- runs the same test suite that IMPALA-11162 added, but with
Hive UDFs created in generic way
- TODO: DDLs and interop with Hive were only tested manually
Change-Id: I692d402784fa91d1256f5d57fa16ac49728447c8
---
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Function.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactory.java
M
fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactoryImpl.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveUdfLoader.java
M fe/src/main/java/org/apache/impala/hive/executor/JavaUdfDataType.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/util/FunctionUtils.java
M
fe/src/test/java/org/apache/impala/hive/executor/HiveLegacyJavaFunctionTest.java
M
fe/src/test/java/org/apache/impala/hive/executor/TestHiveJavaFunctionFactory.java
A
java/test-corrupt-hive-udfs/src/main/java/org/apache/impala/CorruptGenericUdf.java
A
java/test-hive-udfs/src/main/java/org/apache/impala/ClassLoadingErrorHelper.java
A java/test-hive-udfs/src/main/java/org/apache/impala/CorruptGenericUdf.java
A java/test-hive-udfs/src/main/java/org/apache/impala/UnresolvedGenericUdf.java
M testdata/workloads/functional-query/queries/QueryTest/functions-ddl.test
M tests/query_test/test_udfs.py
25 files changed, 511 insertions(+), 63 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/19177/7
--
To view, visit http://gerrit.cloudera.org:8080/19177
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I692d402784fa91d1256f5d57fa16ac49728447c8
Gerrit-Change-Number: 19177
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Daniel Becker <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
