Hi Heekyung, Thanks for the PR. I'm interested in this space and have been working on similar improvements <https://lists.apache.org/thread/66q2941q43rbsp3113nsz9ngm5t9gdqo> a you already saw. I agree that we need better selection criteria for sorted files. In particular, a compaction operation should not deteriorate the clustering. I will review your PR and ping others who might be able to help, even though I'm not a committer.
However, I'm not sure how a user should determine the right value for min-overlap-depth to maintain healthy sortedness. Let's discuss more on the PR. Thanks, Anurag On Thu, Aug 20, 2026 at 7:52 AM heekyung Kim <[email protected]> wrote: > Hi all, > rewrite_data_files selects files by size and delete counts. A table whose > files are all size-healthy but overlap on the sort key is never selected > for sort/zorder compaction: every run reports success and clustering never > improves. There is also no metric that detects this state. Issue #17489 has > a deterministic two-table reproduction — identical data, the size-healthy > copy reports rewritten=0 at overlap depth 8, the fragmented control > rewrites to depth 1. > PR #17504 <https://github.com/apache/iceberg/pull/17504> implements both > halves: > compute_sort_order_stats: a read-only Spark procedure reporting > per-partition overlap depth, computed from manifest lower/upper bounds only > (follows the compute_partition_stats pattern). > min-overlap-depth: an opt-in sort strategy option that rewrites the files > behind that depth. This part follows Peter Vary's suggestion on the PR to > compute the regions where overlap exceeds a threshold and mark those files. > Defaults are unchanged - the option is unset by default and planning reads > bounds only for the sort column, only when set. > CI is green. Peter suggested finding a committer with interest in this > area, hence this mail. > On review load: the PR is three commits, and commit 1 is a self-contained > 283-line core handler (sweep line over bounds, plus tests) - reviewing that > alone validates or kills the approach. Commits 2-3 are thin layers on top > (procedure, planner option). I'm also happy to split it into separate PRs > if that is preferred. > Related work: this composes with the K-way merge strategy proposed in PR > #16305 (detect by overlap, rewrite without shuffle) - I've left review > notes there on how the two could line up. > Any feedback on the direction, naming, or output shape is welcome. > Thanks, Heekyung (gimgit) >
