Csaba Ringhofer has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/24536 )
Change subject: POC: Add GEOMETRY column support for Iceberg and Parquet ...................................................................... POC: Add GEOMETRY column support for Iceberg and Parquet Enable GEOMETRY columns in Iceberg tables when the geospatial_library flag is set to WKB_EXPERIMENTAL. Requires Iceberg format version 3. Geometry values are stored as WKB in Parquet BYTE_ARRAY with GeometryType logical type annotation. Frontend changes: - IcebergSchemaConverter: map GEOMETRY <-> Types.GeometryType, store as "binary" in HMS since it has no native geometry type. - TableDef/AlterTableAddColsStmt: allow GEOMETRY columns for Iceberg v3 tables in WKB_EXPERIMENTAL mode. Backend (Parquet) changes: - Write Parquet GeometryType logical type annotation with crs="OGC:CRS84" so other engines (Trino, Spark) can identify geometry columns. - Skip min/max statistics for geometry columns (WKB byte ordering has no spatial semantics). - Validate geometry CRS on read; reject unsupported CRS. COMPUTE STATS for GEOMETRY columns: - Treated like BINARY: no NDV, but null count and byte size are collected and stored to HMS as BINARY column stats (HMS has no geometry type). - Byte size uses the bytes() builtin for the real WKB size; length() is avoided because it means *spatial* length for geometry (cf. PostGIS ST_Length vs ST_MemSize). Patch Iceberg's HiveSchemaUtil in the shaded jar to handle GEOMETRY/GEOGRAPHY types (maps to "binary"). Without this, every HMS metadata commit (CREATE, INSERT, ALTER) throws UnsupportedOperationException. Limitations: - Always uses default CRS (OGC:CRS84); rejects columns with non-default CRS on read. - Does not write or read bounding rectangle statistics (Parquet geometry covering/stats). - Cannot read Parquet files written by Trino because its native writer uses DELTA_LENGTH_BYTE_ARRAY encoding which Impala does not yet support. Tests: - Unit test for patched HiveSchemaUtil conversion. - Integration tests: CREATE, INSERT, SELECT round-trip, ALTER TABLE ADD/DROP COLUMNS, ST_* function usage. - COMPUTE STATS reports the real WKB byte size for a geometry column (MAX/AVG SIZE via bytes()). - Hive interop test confirming Hive rejects v3 tables. - Manual Trino interop validated: Impala writes geometry via HiveCatalog, Trino reads it successfully with ST_AsText(). Assisted-by: Claude Opus 4.6 Change-Id: I32d8c12a6b71708646fbfc06a6dd8f7aaf4f5e6b --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-metadata-utils.cc M common/thrift/parquet.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java A fe/src/test/java/org/apache/impala/util/IcebergHiveSchemaUtilTest.java M java/shaded-deps/impala-iceberg-runtime/pom.xml A java/shaded-deps/impala-iceberg-runtime/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry.test M tests/custom_cluster/test_geospatial_library.py 16 files changed, 1,082 insertions(+), 138 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/24536/4 -- To view, visit http://gerrit.cloudera.org:8080/24536 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32d8c12a6b71708646fbfc06a6dd8f7aaf4f5e6b Gerrit-Change-Number: 24536 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Balazs Hevele <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
