dwsmith1983 opened a new issue, #18809: URL: https://github.com/apache/iceberg/issues/18809
### Feature Request / Improvement I've been working on data layout for lakehouse tables on my own, comparing layout strategies head to head on TPC-DS store_sales and NYC taxi data. For Iceberg, what made the biggest difference wasn't the curve itself but how each column gets onto it. Today `zorder(...)` and `hilbert(...)` put each column's full ordered bytes into the key. A long like `ss_store_sk` (1 to 100) has all-zero leading bytes, while a double like `ss_sales_price` varies in its sign and exponent bytes. Price takes the high bits, so the rewrite behaves like a sort on price, and queries that filter on the other columns read nearly every file. In my study, clustering on hand-built 16-bit helper columns (each column scaled linearly to its own range) instead of the raw columns cut store_sales box queries from 0.101 to 0.019 of rows read through Iceberg's reader, and from 164 ms to 85 ms. Asking people to maintain helper columns or run an outside tool isn't a great answer when the rewrite already has what it needs, since file stats carry each column's min and max. So it made more sense to put this in the rewrite itself. Others have asked for something similar: #5663, and in #3966 RussellSpitzer suggested `toOrderedBinary` take min/max parameters "so we can shift and tighten the domain". The Javadoc on `SparkHilbertFileRewriteRunner.BITS_PER_COLUMN` also leaves narrower per-column widths open. **Proposal** An opt-in `quantization-bits` option for `zorder` and `hilbert` rewrites: 0 (default, off), or 8, 16, 24 or 32. When set, each clustered column is mapped linearly from its rewrite group's min and max, taken from file bounds, onto that many bits before interleaving, so every column carries the same weight. There's no extra pass over the data and no spec change. With the option off, output is unchanged. **Results** I measured 16 bits against main on one machine: store_sales at SF10 and NYC yellow taxi 2023, the change backported to Spark 4.1, Iceberg's reader, warm cache, 20 queries per family. Each cell is the median fraction of files scanned / ms. store_sales: | query family | main hilbert | hilbert, 16 bits | main zorder | zorder, 16 bits | |---|---|---|---|---| | boxes (item, store, price) | 0.129 / 163 | 0.051 / 77 | 0.103 / 159 | 0.085 / 99 | | store only | 0.971 / 331 | 0.275 / 170 | 0.971 / 314 | 0.239 / 129 | | item only | 0.993 / 292 | 0.239 / 158 | 1.000 / 266 | 0.239 / 162 | | price only | 0.071 / 96 | 0.587 / 218 | 0.044 / * | 0.704 / 224 | \* the timed runs for this one didn't complete; files scanned is from the warm-up round. taxi: | query family | main hilbert | hilbert, 16 bits | main zorder | zorder, 16 bits | |---|---|---|---|---| | boxes (time, distance, fare) | 0.197 / 121 | 0.132 / 110 | 0.192 / 114 | 0.132 / 122 | | time and distance | 0.237 / 153 | 0.158 / 112 | 0.244 / 145 | 0.132 / 113 | | time only | 1.000 / 324 | 0.158 / 133 | 1.000 / 298 | 0.132 / 132 | | distance only | 0.237 / 165 | 1.000 / 291 | 0.256 / 171 | 1.000 / 278 | | fare only | 0.355 / 213 | 1.000 / 280 | 0.462 / 211 | 1.000 / 261 | The column main happened to favor gets worse: price on store_sales, distance and fare on taxi. Taxi also shows the limit of min/max ranges. Its files keep outliers (trip distance up to 345,729, fares up to 386,984), so 99.9% of distances fall in 6 of 65,536 buckets and the key ends up mostly pickup time. That depends on the table, so like choosing Z-order or Hilbert in the first place, it's something you'd turn on per table. Other things I checked: - With the option off, every data file is byte-identical to main's (same sha256 for all files on both tables). - Rewrites with 16 bits ran at 0.72x to 0.83x of main's wall time (median of three builds). - 16-bit tables came out 3% to 10% larger at the same target file size. GGraziadei's #17893 refactors the same rewrite runners. I'm happy to rebase on whichever lands first. ### Query engine Spark ### Willingness to contribute - [X] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
