Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24533
Change subject: POC: Copy Hive ESRI geospatial UDFs into Impala and add WKB mode ...................................................................... POC: Copy Hive ESRI geospatial UDFs into Impala and add WKB mode Copy the ST_* geospatial functions from Hive's org.apache.hadoop.hive.ql.udf.esri package into Impala under org.apache.impala.hive.geospatial.esri. This decouples Impala from Hive's geospatial code and allows independent modifications. The goal is to be able to do modifications to accept WKB instead of ESRI shape binary format. This was a simple change that only affected GeometryUtils.java, which chooses serialization format based on a static member. All other files from Hive (ST_*) are simply copies with exception of changing package names. The copy was based on fresh upstream Hive: https://github.com/apache/hive/commit/ada31342bbd6465c8fa497cd095cebbeaf291b4c This brough a few Hive changes compared to CDP dependency: - HIVE-29323: ST_ConvexHull no longer returns MultiPolygon - HIVE-29352: New ST_DistanceSphere function for great-circle distance between points After the change serialization format can be choosen at startup. Added WKB_EXPERIMENTAL as a new geospatial_library flag value. When enabled, geometries are serialized as OGC Well Known Binary instead of the ESRI Shape format. Key behavioral differences in WKB mode: 1. Native C++ functions are disabled (Java-only path) 2. Relational functions (ST_Intersects, ST_Contains, etc.) only accept (BINARY, BINARY) arguments; STRING overloads removed 3. SRID is not preserved through serialization roundtrips 1. Will be improved in later commit - adding c++ support for WKB seems easier than for ESRI shape, there are few open source projects that did this. 2. Is an intended change - later similar syntax may be possible by adding GEOMETRY type and allowing implicit cast from STRING to GEOMETRY (interpreted as WKT). 3. Is inherent to WKB as it can't store SRID. This will be possibly fixed by using header+WKB format for serialization. Changes: - Copy ~90 ST_* Java files with package rename - Extend GeometryUtils with WKB serialization/deserialization - Add WKB_EXPERIMENTAL to TGeospatialLibrary thrift enum and wire through C++ flag handling and BuiltinsDb - Update gen_geospatial_udf_wrappers.py for new package - Register ST_DistanceSphere and add tests - Split tests: shared tests in geospatial-esri.test and geospatial-esri-extra.test run in both modes; SRID-dependent tests in geospatial-esri-srid.test and STRING overload tests in geospatial-esri-relation-overloads.test run HIVE_ESRI only - Add custom cluster test for WKB_EXPERIMENTAL mode TODOs: - WKB mode has a few commented out tests that need more investigation. - ST_ functions without any tests are collected in an .md file. This is a preexisting issue. I would fix these in a later commit - the goal is mainly to avoid any modification in ST_ files to make changes compared to Hive visible in git history. Change-Id: Ie6e673db4ab6fe4adb64c07f536264d5e847f1a7 --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/function-registry/gen_geospatial_udf_wrappers.py M common/thrift/BackendGflags.thrift M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/BinUtils.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/GeometryUtils.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/Haversine.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/HiveGeometry.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/HiveGeometryOIHelper.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/LogUtils.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/README.md A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Area.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsBinary.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsGeoJson.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsJson.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsShape.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_AsText.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Bin.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_BinEnvelope.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Boundary.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Buffer.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Centroid.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Contains.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_ConvexHull.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_CoordDim.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Crosses.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Difference.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Dimension.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Disjoint.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Distance.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_DistanceSphere.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_EndPoint.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_EnvIntersects.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Envelope.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Equals.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_ExteriorRing.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeodesicLengthWGS84.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomCollection.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromGeoJson.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromJson.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromShape.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromText.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeomFromWKB.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Geometry.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryAccessor.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryN.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryProcessing.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryRelational.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_GeometryType.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_InteriorRingN.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Intersection.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Intersects.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Is3D.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsClosed.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsEmpty.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsMeasured.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsRing.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_IsSimple.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Length.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_LineFromWKB.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_LineString.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_M.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MLineFromWKB.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MPointFromWKB.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MPolyFromWKB.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxM.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxX.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxY.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MaxZ.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinM.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinX.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinY.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MinZ.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MultiLineString.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MultiPoint.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_MultiPolygon.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_NumGeometries.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_NumInteriorRing.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_NumPoints.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Overlaps.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Point.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PointFromWKB.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PointN.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PointZ.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_PolyFromWKB.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Polygon.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Relate.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_SRID.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_SetSRID.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_StartPoint.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_SymmetricDiff.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Touches.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Union.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Within.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_X.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Y.java A fe/src/main/java/org/apache/impala/hive/geospatial/esri/ST_Z.java M java/shaded-deps/hive-exec/pom.xml M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-extra.test A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-relation-overloads.test A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri-srid.test M testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test M tests/custom_cluster/test_geospatial_library.py M tests/query_test/test_geospatial_functions.py 106 files changed, 7,951 insertions(+), 352 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/24533/1 -- To view, visit http://gerrit.cloudera.org:8080/24533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie6e673db4ab6fe4adb64c07f536264d5e847f1a7 Gerrit-Change-Number: 24533 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]>
