Csaba Ringhofer has uploaded a new patch set (#18). ( http://gerrit.cloudera.org:8080/24534 )
Change subject: IMPALA-15161: Add GEOMETRY type + make WKB_EXPERIMENTAL default ...................................................................... IMPALA-15161: Add GEOMETRY type + make WKB_EXPERIMENTAL default Add GEOMETRY as a new primitive type so that geospatial functions in WKB_EXPERIMENTAL mode use typed signatures (e.g. st_point returns GEOMETRY, st_x accepts GEOMETRY) instead of raw BINARY. HIVE_ESRI mode is unchanged, all functions keep their BINARY signature in that case.. The default is changed to WKB_EXPERIMENTAL - before it was NONE, but overriden to HIVE_ESRI in test cluster startup to make testing easier. The change is done here to allow meaningful FE tests for GEOMETRY (otherwise no function would accept/return GEOMTRY): production + BE/FE tests: NONE->WKB_EXPERIMENTAL EE tests: HIVE_ESRI->WKB_EXPERIMENTAL As users who relied on HIVE_ESRI had to specify it in command line flag, it will still override the default lib and keep old behavior. Type system: - Parser: added GEOMETRY as non-reserved keyword - Frontend: proper distinct Type - Backend: minimal cange, piggyback on TYPE_STRING - Block any CAST to/from GEOMETRY - later STRING (WKT) and BINARY (WKB) casts may be added. - Reject GEOMETRY columns in CREATE TABLE - Don't allow returning to client in select list (IMPALA-15244) GEOMETRY in generic expression contexts: - No comparison (<, =) operators - Conditional functions coalesce/if/isnull/ifnull/nvl work on GEOMETRY - Allow a bytes() builtin returning the byte length (currently WKB length) - Can be used in analyical functions (e.g. last), also added for BINARY where it was unsupported without reason Metastore integration: - HMS doesn't have GEOMETRY column type, so a GEOMETRY column is reported to HMS as "binary" while Impala keeps GEOMETRY internally (CatalogOpExecutor.buildFieldSchemaList). This lets a CREATE VIEW expose a GEOMETRY column; the real type is recomputed from the view query when the view is used. Tests: - Usual FE tests for new types - Minimal BE tests bacause Java UDFs can't be called from exprs-test.cc - Minimal EE tests - more meaningful test will be added with Iceberg table integration - Swapped role test_geospatial_library.py and test_geospatial_functions.py due to new default lib Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27 --- M be/src/common/global-flags.cc M be/src/exec/kudu/kudu-util-ir.cc M be/src/exprs/expr-test.cc M be/src/runtime/types.cc M be/src/runtime/types.h M bin/start-impala-cluster.py M common/cup/sql-parser.cup M common/function-registry/impala_functions.py M common/thrift/Types.thrift M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/InPredicate.java M fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/Function.java A fe/src/main/java/org/apache/impala/catalog/GeometryCompatibility.java M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java M fe/src/main/java/org/apache/impala/hive/executor/JavaUdfDataType.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M java/hive-geospatial-functions/src/main/java/org/apache/impala/hive/geospatial/esri/README.md M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test A testdata/workloads/functional-query/queries/QueryTest/geometry-type.test M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test M testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test M testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test M tests/custom_cluster/test_geospatial_library.py M tests/hs2/test_hs2.py M tests/query_test/test_geospatial_functions.py 47 files changed, 703 insertions(+), 232 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/24534/18 -- To view, visit http://gerrit.cloudera.org:8080/24534 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27 Gerrit-Change-Number: 24534 Gerrit-PatchSet: 18 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Aleksandr Efimov <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Balazs Hevele <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]>
