Ankitkj1999 opened a new issue, #18940:
URL: https://github.com/apache/iceberg/issues/18940

   ### Apache Iceberg version
   
   main (commit 234d557)
   
   ### Query engine
   
   All
   
   ### Please describe the bug 
   
   In `ProjectionUtil.java`, projection transforms for `truncate`, `date`, and 
`timestamp` partitions evaluate open comparison predicates (`<`, `>`, `<=`, 
`>=`) by shifting the boundary by 1 before applying the transform:
   
   ```java
   // truncateInteger (Inclusive)
   case LT:
     return predicate(Expression.Operation.LT_EQ, name, 
transform.apply(boundary - 1));
   case GT:
     return predicate(Expression.Operation.GT_EQ, name, 
transform.apply(boundary + 1));
   
   // truncateIntegerStrict (Strict)
   case LT_EQ:
     return predicate(Expression.Operation.LT, name, transform.apply(boundary + 
1));
   case GT_EQ:
     return predicate(Expression.Operation.GT, name, transform.apply(boundary - 
1));
   ```
   
   Similar adjustments exist in `truncateLong` and `truncateLongStrict`.
   
   When `boundary` is at integer/long limits:
   - `boundary - 1` for `Integer.MIN_VALUE` wraps around to `Integer.MAX_VALUE` 
(+2147483647).
   - `boundary + 1` for `Integer.MAX_VALUE` wraps around to `Integer.MIN_VALUE` 
(-2147483648).
   - Identical overflows occur with `Long.MIN_VALUE` and `Long.MAX_VALUE`.
   
   ### Impact
   
   1. **Partition Pruning Bypass (Inclusive)**:
      A query filtering `WHERE id < Integer.MIN_VALUE` (e.g. from optimizer 
domain inference or range bounds) is unsatisfiable. Due to overflow, 
`transform.apply(Integer.MIN_VALUE - 1)` evaluates 
`transform.apply(2147483647)` (e.g. `2147483640` for width 10), producing a 
partition predicate `id_trunc <= 2147483640`. As a result, partition pruning is 
completely bypassed, reading all manifests and scheduling scans for all data 
files even though 0 rows match.
   
   2. **Residual Filter Retainment (Strict)**:
      A filter `WHERE id <= Integer.MAX_VALUE` should match all partitions 
strictly, allowing engines to eliminate residual predicate evaluation on data 
files. Due to overflow (`Integer.MAX_VALUE + 1` wrapping to 
`Integer.MIN_VALUE`), the strict projection yields `id_trunc < -2147483648`, 
causing strict projection to fail for every partition.
   
   ### Reproduction
   
   ```java
   PartitionSpec spec = PartitionSpec.builderFor(new Schema(optional(1, 
"value", Types.IntegerType.get())))
       .truncate("value", 10)
       .build();
   
   // Evaluates to ref(name="value_trunc") <= 2147483640 instead of 
unsatisfiable/null
   Expression proj = Projections.inclusive(spec).project(lessThan("value", 
Integer.MIN_VALUE));
   ```
   
   ### Proposed Fix
   
   In `ProjectionUtil.java`, check for boundary limits before decrementing or 
incrementing:
   - Return `null` for `LT` when `boundary == Integer.MIN_VALUE` / 
`Long.MIN_VALUE`.
   - Return `null` for `GT` when `boundary == Integer.MAX_VALUE` / 
`Long.MAX_VALUE`.
   - Return `null` for `LT_EQ` in strict mode when `boundary == 
Integer.MAX_VALUE` / `Long.MAX_VALUE`.
   - Return `null` for `GT_EQ` in strict mode when `boundary == 
Integer.MIN_VALUE` / `Long.MIN_VALUE`.
   
   Returning `null` allows the projection visitors 
(`InclusiveMetricsEvaluator`, `InclusiveManifestEvaluator`, 
`StrictMetricsEvaluator`, etc.) to safely treat the predicate according to 
standard Iceberg projection semantics.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to