dwsmith1983 opened a new pull request, #18810: URL: https://github.com/apache/iceberg/pull/18810
Z-order and Hilbert rewrites put each column's full ordered bytes into the curve key, so one column usually ends up deciding the order. This adds an opt-in `quantization-bits` option (8, 16, 24 or 32) that scales each clustered column from its rewrite group's min and max, read from file bounds, onto that many bits before interleaving. Clustered columns need bounds (metrics mode `truncate` or `full`), and string columns are rejected. With the option off, keys are unchanged. Hilbert now keeps the schema's spelling of column names, as Z-order already does, so bounds can be found under case-insensitive resolution. Spark 4.2 only for now; backports will follow. Closes #18809 **Test plan** - Core: the byte mapping preserves order, puts min and max in the first and last bucket, clamps out-of-range values, and rejects bad bit widths and ranges. The planner attaches column stats only when a subclass asks for them. - Spark, ranges: bounds are unioned across files, and files that are all null or NaN or predate the column are skipped. A file with values but no bounds, or an infinite bound, fails the group. Dates are read as days and timestamps as micros. - Spark, rewrites: with the option on, Z-order and Hilbert need fewer files for a narrow-range column, each partition uses its own range, rows with nulls are kept, and date, timestamp and timestamp_ntz columns cluster. Invalid bit widths and string columns fail. - With the option off, existing rewrite tests pass unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
