Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24533
to look at the new patch set (#2).
Change subject: POC: Copy Hive ESRI geospatial UDFs into Impala and add WKB mode
......................................................................
POC: Copy Hive ESRI geospatial UDFs into Impala and add WKB mode
Copy the ST_* geospatial functions from Hive's
org.apache.hadoop.hive.ql.udf.esri package into Impala under
org.apache.impala.hive.geospatial.esri. This decouples Impala
from Hive's geospatial code and allows independent modifications.
The goal is to be able to do modifications to accept WKB
instead of ESRI shape binary format. This was a simple
change that only affected GeometryUtils.java, which chooses
serialization format based on a static member. All other files
from Hive (ST_*) are simply copies with exception of changing
package names.
The copy was based on fresh upstream Hive:
https://github.com/apache/hive/commit/ada31342bbd6465c8fa497cd095cebbeaf291b4c
This brough a few Hive changes compared to CDP dependency:
- HIVE-29323: ST_ConvexHull no longer returns MultiPolygon
- HIVE-29352: New ST_DistanceSphere function for great-circle
distance between points
The geometry library that does the actual work (esri-geometry-api)
is added as direct dependency instead of picking it up through Hive.
After the change serialization format can be choosen at startup.
Added WKB_EXPERIMENTAL as a new geospatial_library flag value.
When enabled, geometries are serialized as OGC Well Known Binary
instead of the ESRI Shape format. Key behavioral differences in
WKB mode:
1. Native C++ functions are disabled (Java-only path)
2. Relational functions (ST_Intersects, ST_Contains, etc.) only
accept (BINARY, BINARY) arguments; STRING overloads removed
3. SRID is not preserved through serialization roundtrips
1. Will be improved in later commits - adding c++ support for
WKB seems easier than for ESRI shape, there are some open
source projects that did this.
2. Is an intended change - later similar syntax may be possible
by adding GEOMETRY type and allowing implicit cast from STRING
to GEOMETRY (interpreted as WKT).
3. Is inherent to WKB as it can't store SRID. This will be
possibly fixed by using header+WKB format for serialization.
Changes:
- Copy ~90 ST_* Java files with package rename
- Extend GeometryUtils with WKB serialization/deserialization
- Add WKB_EXPERIMENTAL to TGeospatialLibrary thrift enum and
wire through C++ flag handling and BuiltinsDb
- Update gen_geospatial_udf_wrappers.py for new package
- Register ST_DistanceSphere and add tests
- Split tests: shared tests in geospatial-esri.test and
geospatial-esri-extra.test run in both modes; SRID-dependent
tests in geospatial-esri-srid.test and STRING overload tests
in geospatial-esri-relation-overloads.test run HIVE_ESRI only
- Add custom cluster test for WKB_EXPERIMENTAL mode
TODOs:
- WKB mode has a few commented out tests that need more
investigation.
- ST_ functions without any tests are collected in an .md
file. This is a preexisting issue.
I would fix these in a later commit - the goal is mainly to
avoid any modification in ST_ files to make changes compared
to Hive visible in git history.
Assisted-by: Claude Opus 4.8
Change-Id: Ie6e673db4ab6fe4adb64c07f536264d5e847f1a7
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M common/function-registry/gen_geospatial_udf_wrappers.py
M common/thrift/BackendGflags.thrift
M fe/pom.xml
M
fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/BinUtils.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/GeometryUtils.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/Haversine.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/HiveGeometry.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/HiveGeometryOIHelper.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/LogUtils.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/README.md
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Area.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsBinary.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsGeoJson.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsJson.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsShape.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsText.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Bin.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_BinEnvelope.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Boundary.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Buffer.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Centroid.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Contains.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_ConvexHull.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_CoordDim.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Crosses.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Difference.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Dimension.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Disjoint.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Distance.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_DistanceSphere.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_EndPoint.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_EnvIntersects.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Envelope.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Equals.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_ExteriorRing.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeodesicLengthWGS84.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomCollection.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromGeoJson.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromJson.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromShape.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromText.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromWKB.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Geometry.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryAccessor.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryN.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryProcessing.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryRelational.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryType.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_InteriorRingN.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Intersection.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Intersects.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Is3D.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsClosed.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsEmpty.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsMeasured.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsRing.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsSimple.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Length.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_LineFromWKB.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_LineString.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_M.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MLineFromWKB.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MPointFromWKB.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MPolyFromWKB.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxM.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxX.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxY.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxZ.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinM.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinX.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinY.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinZ.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MultiLineString.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MultiPoint.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MultiPolygon.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_NumGeometries.java
A
fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_NumInteriorRing.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_NumPoints.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Overlaps.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Point.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PointFromWKB.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PointN.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PointZ.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PolyFromWKB.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Polygon.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Relate.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_SRID.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_SetSRID.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_StartPoint.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_SymmetricDiff.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Touches.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Union.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Within.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_X.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Y.java
A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Z.java
M java/shaded-deps/hive-exec/pom.xml
M
testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-extra.test
A
testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-relation-overloads.test
A
testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-srid.test
M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test
M tests/custom_cluster/test_geospatial_library.py
M tests/query_test/test_geospatial_functions.py
107 files changed, 7,957 insertions(+), 352 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/24533/2
--
To view, visit http://gerrit.cloudera.org:8080/24533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie6e673db4ab6fe4adb64c07f536264d5e847f1a7
Gerrit-Change-Number: 24533
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
