nssalian opened a new pull request, #2375: URL: https://github.com/apache/iceberg-go/pull/2375
### Rationale for the change Shredding inference picks a field's `typed_value` type from the first rows of each file. When those rows only hold small integers, the field gets an `INT(8)` or `INT(16)` column, and any later value that doesn't fit falls back to the untyped `value`. One such row is enough to drop the field's bounds for the whole file, so filters on that field can't skip it. On a table clustered by an integer field this happens to the file whose sampled rows fall below 32768: a `duration > 290000` filter over 64 clustered files kept 4 files instead of 3. ### Changes This change infers `int8`, `int16` and `int32` fields as `int32`. The shredding spec stores all three as physical `INT32`, so file size doesn't change, and `VariantEncoding.md` puts every integer width in one "Exact Numeric" equivalence class, so the values are logically unchanged. Inference has widened mixed-width integer fields to the widest observed member since #1351, and arrow-go already puts any integer that fits into a wider typed column; this applies the same result when the sample happens to hold only small values. Readers that rebuild values from the typed column's annotation, such as Java, will see these values as int32 instead of int8/int16; the Exact Numeric class treats them as equal. Java's writer emits the narrow width and also routes narrower values to the residual on an exact type match, so the two writers already differ here. ### Test - `TestAnalyzeSmallIntegersShredAsInt32` covers the analyzer - `TestShreddedVariantWriteSmallSampleKeepsWideBound` writes a file whose sampled rows are 1-4 and whose later rows are 40000-40003, then checks the field keeps its lower and upper bounds. Both fail without the fix (the bounds come back empty). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
