Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24536
Change subject: WIP: Add GEOMETRY column support for Iceberg and Parquet ...................................................................... WIP: Add GEOMETRY column support for Iceberg and Parquet Enable GEOMETRY columns in Iceberg tables when the geospatial_library flag is set to WKB_EXPERIMENTAL. Requires Iceberg format version 3. Geometry values are stored as WKB in Parquet BYTE_ARRAY with GeometryType logical type annotation. Frontend changes: - IcebergSchemaConverter: map GEOMETRY <-> Types.GeometryType, store as "binary" in HMS since it has no native geometry type. - TableDef/AlterTableAddColsStmt: allow GEOMETRY columns for Iceberg v3 tables in WKB_EXPERIMENTAL mode. Backend (Parquet) changes: - Write Parquet GeometryType logical type annotation with crs="OGC:CRS84" so other engines (Trino, Spark) can identify geometry columns. - Skip min/max statistics for geometry columns (WKB byte ordering has no spatial semantics). - Validate geometry CRS on read; reject unsupported CRS. Patch Iceberg's HiveSchemaUtil in the shaded jar to handle GEOMETRY/GEOGRAPHY types (maps to "binary"). Without this, every HMS metadata commit (CREATE, INSERT, ALTER) throws UnsupportedOperationException. Limitations: - Always uses default CRS (OGC:CRS84); rejects columns with non-default CRS on read. - Does not write or read bounding rectangle statistics (Parquet geometry covering/stats). - Cannot read Parquet files written by Trino because its native writer uses DELTA_LENGTH_BYTE_ARRAY encoding which Impala does not yet support. Tests: - Unit test for patched HiveSchemaUtil conversion. - Integration tests: CREATE, INSERT, SELECT round-trip, ALTER TABLE ADD/DROP COLUMNS, ST_* function usage. - Hive interop test confirming Hive rejects v3 tables. - Manual Trino interop validated: Impala writes geometry via HiveCatalog, Trino reads it successfully with ST_AsText(). Assisted-by: Claude Opus 4.6 Change-Id: I32d8c12a6b71708646fbfc06a6dd8f7aaf4f5e6b --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/parquet-metadata-utils.cc M common/thrift/parquet.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java A fe/src/test/java/org/apache/impala/util/IcebergHiveSchemaUtilTest.java M java/shaded-deps/impala-iceberg-runtime/pom.xml A testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry.test M tests/custom_cluster/test_geospatial_library.py 12 files changed, 831 insertions(+), 150 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/24536/1 -- To view, visit http://gerrit.cloudera.org:8080/24536 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I32d8c12a6b71708646fbfc06a6dd8f7aaf4f5e6b Gerrit-Change-Number: 24536 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer <[email protected]>
