turbolytics opened a new issue, #4403:
URL: https://github.com/apache/iceberg-python/issues/4403
### Apache Iceberg version
0.12.0 (latest release)
### Please describe the bug 🐞
A `truncate` partition loses rows whose value is near the minimum int or
long. `EqualTo`, `In` and range filters such as `LessThan(x, 0)` all miss them.
It reproduces on 0.12.0 and on `main` (33b4ce5).
```python
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import EqualTo, LessThan
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import TruncateTransform
from pyiceberg.types import IntegerType, LongType, NestedField
tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db",
warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")
schema = Schema(NestedField(1, "id", LongType(), required=True),
NestedField(2, "x", IntegerType()))
spec = PartitionSpec(PartitionField(source_id=2, field_id=1000,
transform=TruncateTransform(1000), name="x_trunc"))
t = catalog.create_table("ns.trunc", schema=schema, partition_spec=spec)
rows = [{"id": 0, "x": -(2**31)}, {"id": 1, "x": -(2**31) + 100}, {"id": 2,
"x": 5}]
t.append(pa.Table.from_pylist(rows, schema=schema.as_arrow()))
def ids(row_filter):
return sorted(t.scan(row_filter=row_filter).to_arrow()["id"].to_pylist())
print(sorted(task.file.partition[0] for task in t.scan().plan_files())) #
[0, 2147483296]
print(ids(EqualTo("x", -(2**31)))) # [] -- expected [0]
print(ids(EqualTo("x", -(2**31) + 100))) # [] -- expected [1]
print(ids(LessThan("x", 0))) # [] -- expected [0, 1]
```
For `truncate[1000]` on an int, every value from -2147483648 to -2147483001
is affected. A long column behaves the same way near `-(2**63)`.
**Cause.** The writer computes the partition value with pyiceberg-core. Its
math wraps around, as Java's
[`TruncateUtil.truncateInt`](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/util/TruncateUtil.java#L55-L61)
does. So these rows land in partition `2147483296`, a large positive value.
Scan planning projects the filter with the Python transform, [`v - v %
self._width`](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/transforms.py#L867),
which does not wrap. `EqualTo(x, -2147483648)` projects to `-2147484000`, so
it never matches the stored value. `LessThan(x, 0)` projects to `x_trunc <=
-1000`, and the stored value is positive. Either way, the scan prunes the file.
**What a fix could look like.**
- The writer should keep its current output. Java writes the wrapped value
too, so existing tables already hold it.
- Wrapping the Python transform like Java fixes `EqualTo` and `In`.
- Range filters need more than that, because the wrapped value does not sort
with the others. Java's projection
([ProjectionUtil.java#L42-L58](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java#L42-L58))
has the same gap. A [comment on
apache/iceberg#12915](https://github.com/apache/iceberg/issues/12915#issuecomment-2880454693)
notes the `Integer.MIN_VALUE` overflow, and that issue is closed.
apache/iceberg#18111 covers a different overflow in the same function.
I'm not sure of the right fix for range filters. It may need a note in the
spec. I can raise it in apache/iceberg if that helps.
### How I found it
I used [Temper](https://github.com/turbolytics/temper) to audit each table
layout against the same scan guarantees. The reference is the same filter
written as plain Python over the appended rows. Temper helped identify this
bug. In
[`MATRIX.md`](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/MATRIX.md),
only the `truncate` layout fails "int and long extremes are found". The tests
are
[here](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/tests/table/test_temper_scan_layout.py).
Once the approach is agreed, I can send a PR for `EqualTo` and `In`, with
regression tests in the existing test files.
I used an AI assistant to run the tests and draft this issue. I verified the
example on 0.12.0 and on `main`, with pyarrow 26.0.1 and Python 3.12.9.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]