GGraziadei commented on PR #16827: URL: https://github.com/apache/iceberg/pull/16827#issuecomment-5095488368
## Benchmarck Ran the end-to-end comparison Z-Order vs Hilbert. Here the report: https://docs.google.com/document/d/1b7TCuotV-zqTwzW0C9hqmcvQntlbuhXPLMmV8RguFTI/edit?usp=sharing TPC-H SF10 lineitem, same source data and clustered columns in both arms, 58 files rewritten into 98. The queries are the stock TPC-H ones with predicates slightly extended to filter on all three multidimensional index columns — the originals lean on single-column filters and wouldn't exercise the case this change targets. Identical queries in both arms, and each one is run 10 times. - Files scanned: 16/98 (Z-order) vs 11/98 (Hilbert) a 31.25% reduction, consistent across all six queries. Planning-time measurement, so it's deterministic rather than timed; the repeats don't affect it. - Rewrite: 169.8 s vs 196.0 s, +15.4% for Hilbert. Breaks even after ~18 query passes. Caveats I'd rather state up front than defend later: Query latency was 12–38% lower for Hilbert, but I wouldn't quote those numbers as evidence. The 10 repeats are what makes that clear: Z-order's coefficient of variation lands at 20–52% against Hilbert's 3–11%, and swap was near-exhausted for the duration of the run. A clustering change shifts the mean, not the variance, so that asymmetry is the platform rather than the algorithm. On most queries the gap sits inside one standard deviation. The pairwise file overlap metric ranks Hilbert worse while the actual scan set ranks it better. Section 5 of the report argues the metric is the unreliable one, but that's an argument rather than a demonstration. On file size: 16 MB, chosen to get enough files for the comparison to be resolvable on a laptop. Worth being precise about what this does and doesn't limit. The report isn't trying to identify an optimal file size, and TPC-H is scale-invariant, scaling the dataset and the file size together (say SF100 at 160 MB) gives the same 98 files over the same key-space distribution, so the pruning ratio should carry over. What wouldn't carry over is holding SF10 fixed and moving to 256 MB files: that leaves a handful of files, and neither curve would have the granularity to express a 31% difference. The measurement is of the algorithms' clustering behaviour, not of a production configuration. I captured file counts but not scanned bytes, easy to add on the next run. Short version: the pruning improvement is real and well measured, the timing improvement is plausible but unproven, and the magnitudes hold for a fixed file count rather than a fixed file size. Let me know if you'd want the scaled run before this progresses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
