Hello Balazs Hevele, Arnab Karmakar, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24534
to look at the new patch set (#14).
Change subject: IMPALA-15161: Add GEOMETRY type + make WKB_EXPERIMENTAL default
......................................................................
IMPALA-15161: Add GEOMETRY type + make WKB_EXPERIMENTAL default
Add GEOMETRY as a new primitive type so that geospatial
functions in WKB_EXPERIMENTAL mode use typed signatures
(e.g. st_point returns GEOMETRY, st_x accepts GEOMETRY)
instead of raw BINARY.
HIVE_ESRI mode is unchanged, all functions keep their
BINARY signature in that case..
The default is changed to WKB_EXPERIMENTAL - before it
was NONE, but overriden to HIVE_ESRI in test cluster
startup to make testing easier. The change is done
here to allow meaningful FE tests for GEOMETRY
(otherwise no function would accept/return GEOMTRY):
production + BE/FE tests: NONE->WKB_EXPERIMENTAL
EE tests: HIVE_ESRI->WKB_EXPERIMENTAL
As users who relied on HIVE_ESRI had to specify it in
command line flag, it will still override the default
lib and keep old behavior.
Type system:
- Parser: added GEOMETRY as non-reserved keyword
- Frontend: proper distinct Type
- Backend: minimal cange, piggyback on TYPE_STRING
- Block any CAST to/from GEOMETRY - later STRING (WKT)
and BINARY (WKB) casts may be added.
- Reject GEOMETRY columns in CREATE TABLE
- Don't allow returning to client in select list (IMPALA-15244)
GEOMETRY in generic expression contexts:
- No comparison (<, =) operators
- Conditional functions coalesce/if/isnull/ifnull/nvl work
on GEOMETRY
- Allow a bytes() builtin returning the byte length (currently
WKB length)
- Can be used in analyical functions (e.g. last), also added
for BINARY where it was unsupported without reason
Metastore integration:
- HMS doesn't have GEOMETRY column type, so a GEOMETRY column
is reported to HMS as "binary" while Impala keeps GEOMETRY
internally (CatalogOpExecutor.buildFieldSchemaList). This
lets a CREATE VIEW expose a GEOMETRY column; the real type
is recomputed from the view query when the view is used.
Tests:
- Usual FE tests for new types
- Minimal BE tests bacause Java UDFs can't be called from
exprs-test.cc
- Minimal EE tests - more meaningful test will be added
with Iceberg table integration
- Swapped role test_geospatial_library.py and
test_geospatial_functions.py due to new default lib
Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27
---
M be/src/common/global-flags.cc
M be/src/exec/kudu/kudu-util-ir.cc
M be/src/exprs/expr-test.cc
M be/src/runtime/types.cc
M be/src/runtime/types.h
M bin/start-impala-cluster.py
M common/cup/sql-parser.cup
M common/function-registry/impala_functions.py
M common/thrift/Types.thrift
M
fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CastExpr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Function.java
A fe/src/main/java/org/apache/impala/catalog/GeometryCompatibility.java
M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java
M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
M fe/src/main/java/org/apache/impala/catalog/ScalarType.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/JavaUdfDataType.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M
java/hive-geospatial-functions/src/main/java/org/apache/impala/hive/geospatial/esri/README.md
M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test
A testdata/workloads/functional-query/queries/QueryTest/geometry-type.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M
testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test
M testdata/workloads/functional-query/queries/QueryTest/sort-complex.test
M
testdata/workloads/functional-query/queries/QueryTest/struct-in-select-list.test
M tests/custom_cluster/test_geospatial_library.py
M tests/hs2/test_hs2.py
M tests/query_test/test_geospatial_functions.py
47 files changed, 700 insertions(+), 229 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/24534/14
--
To view, visit http://gerrit.cloudera.org:8080/24534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27
Gerrit-Change-Number: 24534
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
