gimgit opened a new pull request, #17504:
URL: https://github.com/apache/iceberg/pull/17504
Relates to #17489.
### What
Adds a read-only Spark procedure `compute_sort_order_stats` that reports,
per partition, how well the data file layout matches the table's declared sort
order:
```sql
CALL catalog.system.compute_sort_order_stats(table => 'db.tbl' [,
snapshot_id => ...]);
```
| output | meaning |
|---|---|
| `max_overlap_depth` | largest number of data files whose ranges on the
first sort field cover a single point — 1 = perfectly clustered |
| `avg_overlap_depth` | mean overlap depth over files with bounds |
| `file_count` / `files_missing_bounds` | files without bounds for the sort
field are excluded from depth and surfaced explicitly |
### Why
As described in #17489, `rewrite_data_files` candidate selection considers
only file size, so a table whose files are all within the healthy size band is
permanently excluded from sort/z-order compaction — the job reports success on
every run while sort-key overlap never improves. Today there is no way to
detect this state: every size-based health metric reports such a table as
perfectly healthy. This was also the missing piece asked in #12761 ("Do you
have a way to identify the non-sorted files somehow?").
This PR intentionally starts with measurement only (no planner/selection
changes), so operators can detect the state and decide when to trigger a
`rewrite-all`/filtered rewrite. A follow-up could add an opt-in selection
option as sketched in #17489.
### How
- `SortOrderStatsHandler` (core): computes overlap depth purely from data
file metadata (`lowerBounds`/`upperBounds` of the first sort field) via a sweep
line over `[lower, upper]` ranges — no data files are opened, nothing is
committed.
- requires a sort order whose first field uses an order-preserving
transform (validated)
- only the first sort field is considered, analogous to how clustering
depth is commonly reported for multi-column layouts
- truncated bounds can only widen a range, so the reported depth is an
upper-bound estimate (documented)
- `ComputeSortOrderStatsProcedure` (Spark 4.1): thin read-only wrapper
returning one row per partition. Scoped to Spark 4.1 first; happy to backport
to 4.0/3.5 if this lands.
- No `Action` layer: the computation is driver-side metadata-only, so the
procedure calls the core handler directly (same pattern as other read-only
procedures). Can add the action layer if preferred.
### Testing
- `TestSortOrderStatsHandler` (core, 10 cases): full overlap, disjoint,
touching boundaries, partial overlap, missing bounds (partial/all), single
file, unsorted table, non-order-preserving transform, `snapshot_id` time travel
- `TestComputeSortOrderStatsProcedure` (spark-extensions, 6 cases × 4
catalogs): empty table, unsorted failure, overlap/disjoint reporting,
per-partition reporting, `snapshot_id`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
