Hello Balazs Hevele, Arnab Karmakar, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24534
to look at the new patch set (#10).
Change subject: POC IMPALA-15161: Add GEOMETRY type for WKB_EXPERIMENTAL mode
and make it defaul
......................................................................
POC IMPALA-15161: Add GEOMETRY type for WKB_EXPERIMENTAL mode and make it defaul
Add GEOMETRY as a new primitive type so that geospatial
functions in WKB_EXPERIMENTAL mode use typed signatures
(e.g. st_point returns GEOMETRY, st_x accepts GEOMETRY)
instead of raw BINARY.
HIVE_ESRI mode is unchanged, all functions keep their
BINARY signature there.
The default is changed to WKB_EXPERIMENTAL - before it
was NONE, but overriden to HIVE_ESRI in test cluster
startup to make testing easier. As users who relied on
HIVE_ESRI had to specify it in command line flag, they
will still use old lib and not break there workloads,
while if it was unset the new mode is used.
Type system:
- Add TPrimitiveType.GEOMETRY in Thrift
- Frontend: distinct Type
- Backend: piggyback on TYPE_STRING
- Block any CAST to/from GEOMETRY
- Reject GEOMETRY columns in CREATE TABLE
- Don't allow returning to client in select list (IMPALA-15244)
GEOMETRY in generic expression contexts:
- No comparison (<, =) operators
- Conditional functions coalesce/if/isnull/ifnull/nvl work
on GEOMETRY
- Add a bytes() builtin returning the byte length (currently
WKB length)
- Can be used in analyical functions (e.g. last), also added
for BINARY where it was unsupported without reason
Metastore integration ("lie to HMS"):
- HMS doesn't have GEOMETRY column type, so a GEOMETRY column
is reported to HMS as "binary" while Impala keeps GEOMETRY
internally (CatalogOpExecutor.buildFieldSchemaList). This
lets a CREATE VIEW expose a GEOMETRY column; the real type
is recomputed from the view query when the view is used.
Tests:
- Usual FE + BE tests for new types
- Swapped role test_geospatial_library.py and
test_geospatial_functions.py due to new default
- Basic EE testing
Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27
---
M be/src/common/global-flags.cc
M be/src/exec/kudu/kudu-util-ir.cc
M be/src/exprs/expr-test.cc
M be/src/runtime/types.cc
M be/src/runtime/types.h
M bin/start-impala-cluster.py
M common/function-registry/impala_functions.py
M common/thrift/Types.thrift
M
fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CastExpr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SetOperationStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Function.java
A fe/src/main/java/org/apache/impala/catalog/GeometryCompatibility.java
M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java
M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
M fe/src/main/java/org/apache/impala/catalog/ScalarType.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
A
fe/src/main/java/org/apache/impala/hive/executor/BinaryToGeometryHiveLegacyFunctionExtractor.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/JavaUdfDataType.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test
A testdata/workloads/functional-query/queries/QueryTest/geometry-type.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
39 files changed, 688 insertions(+), 158 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/24534/10
--
To view, visit http://gerrit.cloudera.org:8080/24534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27
Gerrit-Change-Number: 24534
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
