mhatipog opened a new issue, #19023:
URL: https://github.com/apache/iceberg/issues/19023
### Apache Iceberg version
1.12.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
`truncate(W)` on an int or long column wraps around for values near
`Integer.MIN_VALUE` / `Long.MIN_VALUE`. Partition pruning trusts the wrapped
value, so scans silently drop matching rows. This happens with default
settings, no error is raised.
`truncate(10, -2147483648)` should be `-2147483650`, which does not fit in
an int. `TruncateUtil` computes `value - (((value % width) + width) % width)`
in int arithmetic, so it gives `2147483646`. `truncate(10, -2147483645)` also
gives `2147483646`, and `truncate(10, Long.MIN_VALUE)` gives
`9223372036854775806`. The file holding these rows is recorded in the same
partition as values near `INT_MAX`.
Inclusive projection of a predicate literal near the minimum wraps in the
same way. For example `i > -2147483648` projects to a lower bound near
`INT_MAX`, so almost every file is pruned.
### Reproduction (Spark 4.0.1, iceberg-spark-runtime-4.0_2.13:1.12.0, hadoop
catalog)
```sql
CREATE TABLE hc.db.t (id INT, i INT, l BIGINT) USING iceberg
PARTITIONED BY (truncate(10, i), truncate(10, l));
INSERT INTO hc.db.t VALUES
(0, -2147483648, -9223372036854775808),
(1, -2147483645, -9223372036854775805),
(2, -21, -21),
(3, 5, 5),
(4, 2147483647, 9223372036854775807);
SELECT partition, record_count FROM hc.db.t.files;
-- {i_trunc=2147483646, l_trunc=9223372036854775806} 2 <- ids 0 and 1
(the MIN_VALUE rows)
-- {i_trunc=2147483640, l_trunc=9223372036854775800} 1 <- id 4
(MAX_VALUE, correct)
-- {i_trunc=-30, l_trunc=-30} 1
-- {i_trunc=0, l_trunc=0} 1
```
| Query | Expected ids | Actual ids |
|---|---|---|
| `SELECT id FROM hc.db.t WHERE i < 0` | 0, 1, 2 | 2 |
| `... WHERE i < -21` | 0, 1 | (none) |
| `... WHERE i <= -21` | 0, 1, 2 | 2 |
| `... WHERE i > -2147483648` | 1, 2, 3, 4 | 1 |
| `... WHERE i >= -2147483645` | 1, 2, 3, 4 | 1 |
| `... WHERE i BETWEEN -2147483648 AND -2147483640` | 0, 1 | (none) |
| `... WHERE l < -21` | 0, 1 | (none) |
| `... WHERE l > -9223372036854775808` | 1, 2, 3, 4 | 1 |
`i = -2147483648` and `i IN (-2147483648, 5)` return the right rows in Spark.
### Other implementations
- pyiceberg 0.12.0 writes the same wrapped partition values (2147483646 /
9223372036854775806). Its reader also loses `i = -2147483648`, `i IN
(-2147483648, 5)` and `l = -9223372036854775808`, because its Python transform
of the literal gives the unwrapped `-2147483650`, which does not match the
stored value.
- Trino 484 (Iceberg connector) loses the same rows for `<` / `>` predicates
on these tables.
### Relation to existing issues
- #18111 / #18112 fix an intermediate overflow at width `Integer.MAX_VALUE`,
where the true result fits. That issue explicitly sets aside the case where the
true result does not fit, which is this one.
- #12915 mentioned in a comment that truncating `Integer.MIN_VALUE` can
overflow; it was closed as stale without a fix, and without the wrong-result
effect on scans.
- #17027 is the decimal version (the truncated value exceeds the source
precision; there the write fails rather than returning wrong results).
Possible directions: reject or clamp values whose truncated result is not
representable (e.g. map them to `MIN_VALUE` aligned down as far as fits, and
make projection treat that bucket as unbounded below), or have inclusive
projection never prune a partition whose value is greater than the source value
it was computed from. Either way, the transform and the projection need to
agree, and the spec should say what truncate returns when the result is out of
range.
Found while cross-checking the same tables across several readers.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]