Hello Balazs Hevele, Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24534
to look at the new patch set (#7).
Change subject: POC IMPALA-15161: Add GEOMETRY type for WKB_EXPERIMENTAL mode
......................................................................
POC IMPALA-15161: Add GEOMETRY type for WKB_EXPERIMENTAL mode
Add GEOMETRY as a new primitive type so that geospatial
functions in WKB_EXPERIMENTAL mode use typed signatures
(e.g. st_point returns GEOMETRY, st_x accepts GEOMETRY)
instead of raw BINARY.
Major TODOs:
- disallow order by, group by, etc due to the missing =
and < operators
- rethink client handling - probably the best would be
to have a query option about whether to do WKT
conversion on server side
- probably WKB_EXPERIMENTAL should be default after this
patch, allowing using geometry functions in non custom
cluster tests easily
Type system:
- Add TPrimitiveType.GEOMETRY in Thrift
- Frontend: PrimitiveType, Type, ScalarType, parser
- Backend: piggyback on TYPE_STRING, discriminated by a
StringValSubtype enum {STRING, BINARY, GEOMETRY} on
ColumnType (replacing the earlier is_binary_ bool); add
IsBinaryType()/IsGeometryType()/IsPlainStringType()
- Add GeometryCompatibility to block implicit conversions
- Block CAST to/from GEOMETRY in CastExpr
- Reject GEOMETRY columns in CREATE TABLE
Function registration (WKB_EXPERIMENTAL only):
- Add BinaryToGeometryHiveLegacyFunctionExtractor that maps
BytesWritable to GEOMETRY instead of BINARY
- Update HiveEsriGeospatialBuiltins to use GEOMETRY types
for args/returns when geospatial_library=WKB_EXPERIMENTAL
- Typed boundaries: st_geomfromwkb(BINARY)->GEOMETRY,
st_asbinary(GEOMETRY)->BINARY
- BuiltinsDb skips GEOMETRY when registering aggregate and
analytic builtins (as it already does BINARY), so
min/max/sample/appx_median/first_value are not defined
for GEOMETRY
GEOMETRY in generic expression contexts (parity with BINARY):
- Conditional functions coalesce/if/isnull/ifnull/nvl work
on GEOMETRY (they only pass an operand through, so the
lack of comparison/ordering does not matter)
- Add a bytes() builtin returning the byte length of a
STRING/BINARY/GEOMETRY value (e.g. the WKB byte size)
Metastore integration ("lie to HMS"):
- HMS rejects a "geometry" column type, so a GEOMETRY column
is reported to HMS as "binary" while Impala keeps GEOMETRY
internally (CatalogOpExecutor.buildFieldSchemaList). This
lets a CREATE VIEW expose a GEOMETRY column; the real type
is recomputed from the view query when the view is used.
Runtime fixes:
- LiteralExpr: handle GEOMETRY in constant folding
- HiveGenericJavaFunction: map GEOMETRY to Hive's BINARY
PrimitiveCategory and TypeInfo
- JavaUdfDataType: accept GEOMETRY for BytesWritable compat
- query-result-set.cc, hs2-util.cc: serialize GEOMETRY
columns as binary data with BINARY_TYPE descriptor
- Byte-array write paths assert the value is a plain STRING
or BINARY (IsPlainStringType() || IsBinaryType()), not a
special subtype such as GEOMETRY
Tests:
- AnalyzeExprsTest/AnalyzeStmtsTest: GEOMETRY behavior in
casts, comparisons, conditional functions, aggregates,
analytic functions, set operations and inline views,
grouped with each feature's existing tests
- AnalyzeDDLTest: CREATE TABLE rejected, CREATE VIEW allowed
- test_geospatial_library.py + QueryTest/geometry-type.test:
st_point returns GEOMETRY in WKB mode; bytes() and a
GEOMETRY view work end to end
- Relax geospatial-esri.test CATCH patterns for WKB mode
HIVE_ESRI mode is unchanged; all existing ST_ functions
keep their BINARY signatures there.
Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27
---
M be/src/exec/kudu/kudu-util-ir.cc
M be/src/exprs/expr-test.cc
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/hs2-util.cc
M be/src/service/query-result-set.cc
M common/function-registry/impala_functions.py
M common/thrift/Types.thrift
M
fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CastExpr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Function.java
A fe/src/main/java/org/apache/impala/catalog/GeometryCompatibility.java
M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java
M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
M fe/src/main/java/org/apache/impala/catalog/ScalarType.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
A
fe/src/main/java/org/apache/impala/hive/executor/BinaryToGeometryHiveLegacyFunctionExtractor.java
M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java
M fe/src/main/java/org/apache/impala/hive/executor/JavaUdfDataType.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
A testdata/workloads/functional-query/queries/QueryTest/geometry-type.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
34 files changed, 409 insertions(+), 63 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/24534/7
--
To view, visit http://gerrit.cloudera.org:8080/24534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0ff29cb15ab45f3899bf453041e605a32f195c27
Gerrit-Change-Number: 24534
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
