GGraziadei opened a new issue, #17805:
URL: https://github.com/apache/iceberg/issues/17805
Title: Spark: Backport Hilbert-curve sort compaction to Spark 4.0 and 3.5
## Description
Hilbert-curve ordering for `rewriteDataFiles` compaction is currently
implemented only for **Spark 4.1**. The shared building blocks already live
in the version-agnostic `core` module, so the remaining work is to port the
Spark integration to the still-supported **Spark 4.0** and **Spark 3.5**
modules to keep feature parity across Spark versions.
Hilbert ordering, like Z-order, is an engine-side compaction strategy: it
sorts data files along a space-filling curve for better multi-column data
locality. It is **not** persisted in table metadata, so this is purely an
engine-integration backport with no table-format or format-version impact.
## Current state
Already present and shared (no change needed):
- `core`: `org.apache.iceberg.expressions.Hilbert` (the `Term`)
- `core`: `org.apache.iceberg.util.HilbertByteUtils`
Implemented only in Spark 4.1:
- New: `spark/v4.1/.../actions/SparkHilbertUDF.java`
- New: `spark/v4.1/.../actions/SparkHilbertFileRewriteRunner.java`
- Wired: `spark/v4.1/.../actions/RewriteDataFilesSparkAction.java`
- Wired: `spark/v4.1/.../Spark3Util.java` (parses the `hilbert(...)` sort
transform)
- Wired: `spark/v4.1/.../procedures/RewriteDataFilesProcedure.java`
(accepts Hilbert sort expressions; rejects mixing Hilbert with Z-order or
identity sort columns)
Missing in Spark 4.0 and 3.5:
- `grep -r Hilbert spark/v4.0/spark/src/main` → 0 references
- `grep -r Hilbert spark/v3.5/spark/src/main` → 0 references
## Scope of work
For each of `spark/v4.0` and `spark/v3.5`:
1. Port `SparkHilbertUDF` and `SparkHilbertFileRewriteRunner`.
2. Wire `Spark3Util` to parse the `hilbert(...)` transform.
3. Wire `RewriteDataFilesSparkAction` to select the Hilbert runner.
4. Wire `RewriteDataFilesProcedure` to accept Hilbert sort expressions and
keep the existing guardrails (no mixing with Z-order / identity columns).
5. Port the corresponding tests (e.g. the Hilbert cases in
`TestRewriteDataFilesAction` and any Hilbert-specific unit tests).
## Notes / considerations
- Watch for Spark-version API differences (Scala 2.12 vs 2.13 for 3.5;
Connector/Catalyst API drift between 3.5, 4.0, 4.1). The UDF and runner
should port with minimal changes since the heavy lifting is in `core`.
- No format-version gating is required: Hilbert leaves no trace in table
metadata, so old readers/writers are unaffected.
- Keep behavior identical across versions; add tests that assert equivalence
of the produced ordering against the shared `HilbertByteUtils` reference,
mirroring the Z-order test pattern.
## Acceptance criteria
- Hilbert compaction usable from `rewriteDataFiles` action and the
`rewrite_data_files` SQL procedure on Spark 4.0 and 3.5.
- Parity tests pass on all three Spark versions.
- No changes to `core` or table-format spec.
---
*AI Disclosure: this issue was drafted with Claude Opus 4.6 (Claude Code)
and reviewed by a human before filing.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
