C-Loftus opened a new pull request, #2517: URL: https://github.com/apache/iceberg-go/pull/2517
This PR adds support for Hilbert-curve ordering for data file rewrites, similar to the iceberg java spark functionality here: https://github.com/apache/iceberg/pull/16827 - New compaction option `WithCompactionHilbertOrder("geom", ...)` orders each partition of a group along a Hilbert curve over a geometry or geography column. Intended to address #2043 - Rows close in space land in the same file and row group, so per-file geometry bounds overlap little and spatial filters can prune files. - `WithHilbertBits` sets grid resolution (default 16 bits per axis, max 32); `WithHilbertBounds` fixes the extent instead of deriving it from the data. - Cannot be combined with `WithCompactionSortOrder`; rejects non-geo columns, out-of-range bits and bad bounds. - Output files record no `sort_order_id`, since a Hilbert order is not an Iceberg sort order. ## Internal implementation - Rows are read with deletes applied and split by the current partition spec; each partition is ordered on its own. - Each geometry is reduced to the centre of its XY bounding box, snapped to a `2^bits x 2^bits` grid over the partition's extent, and sorted by that cell's distance along the curve. - Rows with no location (null, empty, or only empty members) go last, in scan order. - Ordered rows are handed to the clustered writer with `withPresortedInput`, which stops the writer re-sorting batches by the table's default sort order. - Gather and batching code is shared with the sort strategy: `yieldSortedPartition` was split into `checkCompactionBatches`, `naturalRowRefs` and `yieldOrderedRows`, with no change to the sort algorithm. - Limits: the whole group is held in memory, rows are ordered only within a group, and coordinates are planar (no antimeridian wrapping). ## Decoding - Centres come from a new `internal.WKBCenterXY`, built on the decoder the writer's statistics already use. - Accepts ISO WKB and EWKB, with or without an SRID. - Walks nested geometry collections instead of calling `geom.T.Bounds()`, which panics on them. - Empty members decode to NaN and are skipped, so they don't poison a non-empty geometry's bounds. - No sentinel distance for unlocated rows: every `uint64` is a valid distance on a 32-bit grid, so a `located` flag is compared first. - Cell quantisation clamps endpoints and normalises on halved values, so an extent wider than `MaxFloat64` can't overflow and collapse to cell 0. ## Testing / edge cases I tried to test - Curve: bijection, adjacency, quadrant locality, 32-bit overflow. - Quantisation: edge cases, equal-width cells, extreme ranges. - Centre: byte orders, nested and empty-member collections, EWKB, extreme and non-finite coordinates. - Ordering: nulls last at 32 bits, early-stop release with a checked allocator. - End-to-end through `ExecuteCompactionGroup` and `RewriteDataFiles`: curve order across rolled files with per-file bounds checked against the rows, partitioned tables, geography columns, position deletes, v3 row lineage, and nested/EWKB/empty-member inputs with bin-pack as the passing control. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
