mhatipog opened a new issue, #19023:
URL: https://github.com/apache/iceberg/issues/19023

   ### Apache Iceberg version
   
   1.12.0 (latest release)
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   `truncate(W)` on an int or long column wraps around for values near 
`Integer.MIN_VALUE` / `Long.MIN_VALUE`. Partition pruning trusts the wrapped 
value, so scans silently drop matching rows. This happens with default 
settings, no error is raised.
   
   `truncate(10, -2147483648)` should be `-2147483650`, which does not fit in 
an int. `TruncateUtil` computes `value - (((value % width) + width) % width)` 
in int arithmetic, so it gives `2147483646`. `truncate(10, -2147483645)` also 
gives `2147483646`, and `truncate(10, Long.MIN_VALUE)` gives 
`9223372036854775806`. The file holding these rows is recorded in the same 
partition as values near `INT_MAX`.
   
   Inclusive projection of a predicate literal near the minimum wraps in the 
same way. For example `i > -2147483648` projects to a lower bound near 
`INT_MAX`, so almost every file is pruned.
   
   ### Reproduction (Spark 4.0.1, iceberg-spark-runtime-4.0_2.13:1.12.0, hadoop 
catalog)
   
   ```sql
   CREATE TABLE hc.db.t (id INT, i INT, l BIGINT) USING iceberg
     PARTITIONED BY (truncate(10, i), truncate(10, l));
   
   INSERT INTO hc.db.t VALUES
     (0, -2147483648, -9223372036854775808),
     (1, -2147483645, -9223372036854775805),
     (2, -21, -21),
     (3, 5, 5),
     (4, 2147483647, 9223372036854775807);
   
   SELECT partition, record_count FROM hc.db.t.files;
   -- {i_trunc=2147483646, l_trunc=9223372036854775806}  2   <- ids 0 and 1 
(the MIN_VALUE rows)
   -- {i_trunc=2147483640, l_trunc=9223372036854775800}  1   <- id 4 
(MAX_VALUE, correct)
   -- {i_trunc=-30, l_trunc=-30}                         1
   -- {i_trunc=0, l_trunc=0}                             1
   ```
   
   | Query | Expected ids | Actual ids |
   |---|---|---|
   | `SELECT id FROM hc.db.t WHERE i < 0` | 0, 1, 2 | 2 |
   | `... WHERE i < -21` | 0, 1 | (none) |
   | `... WHERE i <= -21` | 0, 1, 2 | 2 |
   | `... WHERE i > -2147483648` | 1, 2, 3, 4 | 1 |
   | `... WHERE i >= -2147483645` | 1, 2, 3, 4 | 1 |
   | `... WHERE i BETWEEN -2147483648 AND -2147483640` | 0, 1 | (none) |
   | `... WHERE l < -21` | 0, 1 | (none) |
   | `... WHERE l > -9223372036854775808` | 1, 2, 3, 4 | 1 |
   
   `i = -2147483648` and `i IN (-2147483648, 5)` return the right rows in Spark.
   
   ### Other implementations
   
   - pyiceberg 0.12.0 writes the same wrapped partition values (2147483646 / 
9223372036854775806). Its reader also loses `i = -2147483648`, `i IN 
(-2147483648, 5)` and `l = -9223372036854775808`, because its Python transform 
of the literal gives the unwrapped `-2147483650`, which does not match the 
stored value.
   - Trino 484 (Iceberg connector) loses the same rows for `<` / `>` predicates 
on these tables.
   
   ### Relation to existing issues
   
   - #18111 / #18112 fix an intermediate overflow at width `Integer.MAX_VALUE`, 
where the true result fits. That issue explicitly sets aside the case where the 
true result does not fit, which is this one.
   - #12915 mentioned in a comment that truncating `Integer.MIN_VALUE` can 
overflow; it was closed as stale without a fix, and without the wrong-result 
effect on scans.
   - #17027 is the decimal version (the truncated value exceeds the source 
precision; there the write fails rather than returning wrong results).
   
   Possible directions: reject or clamp values whose truncated result is not 
representable (e.g. map them to `MIN_VALUE` aligned down as far as fits, and 
make projection treat that bucket as unbounded below), or have inclusive 
projection never prune a partition whose value is greater than the source value 
it was computed from. Either way, the transform and the projection need to 
agree, and the spec should say what truncate returns when the result is out of 
range.
   
   Found while cross-checking the same tables across several readers.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to