Shahzeb-DataEngineer opened a new issue, #18988: URL: https://github.com/apache/iceberg/issues/18988
### Apache Iceberg version None ### Query engine None ### Please describe the bug 🐞 ## Description With a `decimal(9,2)` column partitioned by `bucket(value, 16)`, the query `WHERE value = 100.0` silently misses rows holding `100.00`, because the literal keeps its written scale when bound but the bucket hash is scale-sensitive. `Literals.DecimalLiteral.to(DECIMAL)` and `StringLiteral.to(DECIMAL)` both say "do not change decimal scale" and return the literal unchanged, while `IntegerLiteral`/`LongLiteral`/`FloatLiteral`/`DoubleLiteral.to(DECIMAL)` all `setScale(column scale, HALF_UP)`. `Binder` converts literals via `Literal.to(type)`, so a decimal literal keeps its written scale. But `BucketUtil.hash(BigDecimal)` hashes the raw unscaled-value bytes, so numerically equal decimals at different scales hash differently. The spec (format/spec.md) says hash results are "not dependent on decimal scale, which is part of the type, not the data value". That holds for stored data, where the column scale is fixed, but not for un-normalized query literals. ## To Reproduce Schema `decimal(9,2)`, partition spec `bucket(value, 16)`: - `value = 100.00` projects to `value_bucket == 8` - `value = 100.0` projects to `value_bucket == 0` A manifest whose stored partition value is 8 is then pruned by `ManifestEvaluator.eq` (lower == upper == 8 vs projected 0 returns ROWS_CANNOT_MATCH), so the rows are silently missing from `WHERE value = 100.0`. The same flaw reaches IN lists via `Bucket.project` -> `ProjectionUtil.transformSet`. Same family at row level: binding `IN(1.1, 2.2)` on `decimal(9,2)`, then `BoundSetPredicate.test(1.100)` returns false, because `Evaluator.in` / `BoundSetPredicate.test` use `Set.contains` on raw BigDecimals and `BigDecimal.equals`/`hashCode` are scale-sensitive (equals false, compareTo == 0, contains false). Residual evaluation drops matching rows. Verified on current main (bc94d8e, 2026-11-16): calibration `BucketUtil.hash(14.20)` = -500754589 matches the spec example. ## Expected behavior Numerically equal decimal literals project to the same bucket regardless of written scale, and IN evaluation matches numerically equal decimals. ## Possible fix Normalize the decimal literal to the column scale at bind time (`DecimalLiteral.to` / `StringLiteral.to`), taking care that values with more fractional digits than the column scale must not be silently rounded for EQ; or scale-normalize inside the bucket projection. Changing `BucketUtil.hash` itself would break spec compatibility with existing data. ## Notes - Closest prior report is #7882 (closed, Hive-side trimming, identity partition); it did not fix this code path, `Set.contains` is still in current main. - I can contribute a fix for this bug independently. ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
