Hello Iceberg developers, I have been following the recent discussions and implementation efforts around native geospatial support in Iceberg, including Geometry/Geography types, bounding box statistics, and spatial pruning.
>From what I have seen so far, the proposed approaches generally rely on storing spatial bounding regions (bounding boxes) as table or manifest metadata so that query engines can prune irrelevant files during planning. Over the past several months, I have been experimenting with a different approach that may be worth discussing. Instead of relying on metadata-based spatial pruning, my approach physically partitions the data using a QuadTree-inspired hierarchy. The idea is to preserve spatial locality through data layout rather than through additional metadata structures. Some potential advantages I have observed are: - It does not require global ordering using space-filling curves such as Hilbert or Z-order. - Query performance is comparable to approaches based on space-filling curves in my experiments. - It naturally supports incremental ingestion. Newly arriving data can be appended without requiring expensive global reordering or large shuffle operations. - This makes it particularly attractive for continuously growing spatial datasets, where maintaining a global ordering can become increasingly expensive. To evaluate the idea, I implemented it on top of three different spatial storage representations: - WKB - Spatial Parquet - Flatten Spatial Parquet The implementation does not modify either Iceberg or Parquet. Instead, it is implemented as an external framework layered on top of the existing systems, making it independent of specific Iceberg or Parquet versions. If such an approach were ever considered for native integration into Iceberg, the implementation details would naturally be different. I would appreciate feedback from the community on whether this direction seems interesting enough for further discussion. In particular, I would be interested in understanding whether the Iceberg community sees value in exploring physical spatial partitioning strategies, in addition to metadata-based spatial pruning. If there is interest, I would be happy to share the implementation details, benchmarks, and the design decisions behind the framework. Thank you.
