Hello Balazs Hevele, Arnab Karmakar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24536
to look at the new patch set (#12).
Change subject: IMPALA-15162: Add GEOMETRY column support for Iceberg and
Parquet
......................................................................
IMPALA-15162: Add GEOMETRY column support for Iceberg and Parquet
Enable GEOMETRY columns in Iceberg tables when the
geospatial_library flag is set to WKB_EXPERIMENTAL (default).
Requires Iceberg format version 3. Geometry values are
stored as WKB in Parquet BYTE_ARRAY with GeometryType
logical type annotation.
Frontend changes:
- IcebergSchemaConverter: map GEOMETRY <-> Types.GeometryType,
store as "binary" in HMS since it has no native geometry type.
- TableDef/AlterTableAddColsStmt: allow GEOMETRY columns for
Iceberg v3 tables in WKB_EXPERIMENTAL mode.
Backend (Parquet) changes:
- Write Parquet GeometryType logical type annotation with
default crs="OGC:CRS84".
- Validate logical type and CRS on Parquet read.
COMPUTE STATS for GEOMETRY columns:
- Treated like BINARY: no NDV, but null count and byte size
are collected and stored to HMS as BINARY column stats
(HMS has no geometry type).
- Byte size uses the bytes() builtin for the real WKB size;
length() is avoided because it means spatial length for
geometry in some engines(PostGIS ST_Length vs ST_MemSize).
Patch Iceberg's HiveSchemaUtil in the shaded jar to handle
GEOMETRY/GEOGRAPHY types (maps to "binary"). Without this,
every HMS metadata commit (CREATE, INSERT, ALTER) throws
UnsupportedOperationException.
Also pushed a PR to fix this in Iceberg:
https://github.com/apache/iceberg/pull/17112
Note that writing "unknown" works with latest Hive dependency,
but breaks Trino interop as it doesn't know about "unknown"
type yet.
Limitations:
- Always uses default CRS (OGC:CRS84); rejects columns with
non-default CRS on read (IMPALA-15169).
- Does not write or read bounding rectangle statistics
(Parquet geometry covering/stats) (see IMPALA-15163)
Tests:
- Unit test for patched HiveSchemaUtil conversion.
- Integration tests: CREATE, INSERT, SELECT round-trip,
ALTER TABLE ADD/DROP COLUMNS, ST_* function usage.
- COMPUTE STATS reports the real WKB byte size for a
geometry column (MAX/AVG SIZE via bytes()).
- Hive interop test confirming Hive rejects GEOMETRY columns.
- Trino interop test: Impala and Trino can read geometry
columns written by the other.
Assisted-by: Claude Opus 4.6
Change-Id: I32d8c12a6b71708646fbfc06a6dd8f7aaf4f5e6b
---
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M bin/jenkins/critique-gerrit-review.py
M fe/src/main/java/org/apache/impala/analysis/AlterTableAddColsStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/IcebergSchemaConverter.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A fe/src/test/java/org/apache/impala/util/IcebergHiveSchemaUtilTest.java
M java/shaded-deps/impala-iceberg-runtime/pom.xml
A
java/shaded-deps/impala-iceberg-runtime/src/main/java/org/apache/iceberg/hive/HiveSchemaUtil.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-geometry.test
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-geometry.test
M tests/custom_cluster/test_iceberg_trino_interop.py
M tests/query_test/test_geospatial_functions.py
20 files changed, 715 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/24536/12
--
To view, visit http://gerrit.cloudera.org:8080/24536
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I32d8c12a6b71708646fbfc06a6dd8f7aaf4f5e6b
Gerrit-Change-Number: 24536
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Balazs Hevele <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
