Peter Rozsa has uploaded a new patch set (#22). ( http://gerrit.cloudera.org:8080/19425 )
Change subject: IMPALA-11745: Add Hive's ESRI geospatial functions as builtins ...................................................................... IMPALA-11745: Add Hive's ESRI geospatial functions as builtins This change adds geospatial functions from Hive's ESRI library as builtin UDFs. Plain Hive UDFs are imported without changes, but the generic and varargs functions are handled differently; generic functions are added with all of the combinations of their parameters (cartesian product of the parameters), and varargs functions are unfolded as an nth parameter simple function. The varargs function wrappers are generated at build time and they can be configured in gen_geospatial_udf_wrappers.py. These additional steps are required because of the limitations in Impala's UDF Executor (lack of varargs support and only partial generics support) which could be further improved; in this case, the additional wrapping/mapping steps could be removed. Changes regarding function handling/creating are sourced from https://gerrit.cloudera.org/c/19177 A new backend flag was added to turn this feature on/off as "geospatial_library". The default value is "NONE" which means no geospatial function gets registered as builtin, "HIVE_ESRI" value enables this implementation. Known limitations: - ST_MultiLineString, ST_MultiPolygon only works with the WKT overload - ST_Polygon supports a maximum of 6 pairs of coordinates - ST_MultiPoint, ST_LineString supports a maximum of 7 pairs of coordinates - ST_ConvexHull, ST_Union supports a maximum of 6 geoms These limits can be increased in gen_geospatial_udf_wrappers.py and HiveEsriGeospatialBuiltins.java Tests: - test_geospatial_udfs.py added based on https://github.com/Esri/spatial-framework-for-hadoop Co-Authored-by: Csaba Ringhofer <[email protected]> Change-Id: If0ca02a70b4ba244778c9db6d14df4423072b225 --- M be/src/common/global-flags.cc M be/src/exprs/hive-udf-call.cc M be/src/util/backend-gflag-util.cc M bin/start-impala-cluster.py M common/function-registry/CMakeLists.txt M common/function-registry/gen_builtins_catalog.py A common/function-registry/gen_geospatial_udf_wrappers.py M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/HiveEsriGeospatialBuiltins.java M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java A fe/src/main/java/org/apache/impala/hive/executor/BinaryToBinaryHiveLegacyFunctionExtractor.java M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactory.java M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactoryImpl.java A fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyFunctionExtractor.java M fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java M fe/src/main/java/org/apache/impala/hive/executor/ImpalaDoubleWritable.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/hive/executor/HiveLegacyJavaFunctionTest.java M fe/src/test/java/org/apache/impala/hive/executor/TestHiveJavaFunctionFactory.java M java/CMakeLists.txt M java/shaded-deps/hive-exec/pom.xml M testdata/datasets/README A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test A tests/custom_cluster/test_geospatial_library.py A tests/query_test/test_geospatial_functions.py 31 files changed, 3,489 insertions(+), 157 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/19425/22 -- To view, visit http://gerrit.cloudera.org:8080/19425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If0ca02a70b4ba244778c9db6d14df4423072b225 Gerrit-Change-Number: 19425 Gerrit-PatchSet: 22 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]>
