turbolytics opened a new issue, #4404:
URL: https://github.com/apache/iceberg-python/issues/4404

   ### Apache Iceberg version
   
   0.12.0 (latest release)
   
   ### Please describe the bug 🐞
   
   Filtered scans raise or return wrong rows at two kinds of edge values. Both 
bugs reproduce on 0.12.0 and on `main` (33b4ce5). A third bug, in `truncate` 
partitions, is in #4403.
   
   ```python
   import datetime as dt
   import math
   import tempfile
   
   import pyarrow as pa
   
   from pyiceberg.catalog.sql import SqlCatalog
   from pyiceberg.expressions import EqualTo, In, LessThanOrEqual, NotEqualTo
   from pyiceberg.partitioning import PartitionField, PartitionSpec
   from pyiceberg.schema import Schema
   from pyiceberg.transforms import YearTransform
   from pyiceberg.types import DateType, FloatType, LongType, NestedField
   
   tmp = tempfile.mkdtemp()
   catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", 
warehouse=f"file://{tmp}/warehouse")
   catalog.create_namespace("ns")
   
   
   def table(name, column, spec, values):
       schema = Schema(NestedField(1, "id", LongType(), required=True), 
NestedField(2, "x", column))
       t = catalog.create_table(f"ns.{name}", schema=schema, 
partition_spec=spec)
       t.append(pa.Table.from_pylist([{"id": i, "x": v} for i, v in 
enumerate(values)], schema=schema.as_arrow()))
       return t
   
   
   def ids(t, row_filter):
       try:
           return 
sorted(t.scan(row_filter=row_filter).to_arrow()["id"].to_pylist())
       except OverflowError as e:
           return repr(e)
   
   
   # 1. year partitions and the last representable date
   spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, 
transform=YearTransform(), name="x_year"))
   t = table("year", DateType(), spec, [dt.date(2024, 5, 1), dt.date(9999, 12, 
31)])
   print(ids(t, LessThanOrEqual("x", dt.date(9999, 12, 31))))  # 
OverflowError('date value out of range') -- expected [0, 1]
   
   # 2. inf in a float column
   t = table("floats", FloatType(), PartitionSpec(), [1.5, math.inf])
   print(ids(t, EqualTo("x", math.inf)))  # [] -- expected [1]
   print(ids(t, NotEqualTo("x", math.inf)))  # [0, 1] -- expected [0]
   print(ids(t, In("x", {math.inf, 1.5})))  # [0] -- expected [0, 1]
   ```
   
   #### 1. `year` and `month` partitions raise `OverflowError` at 9999-12-31 
and 0001-01-01
   
   `LessThanOrEqual` and `GreaterThan` raise at 9999-12-31. 
`GreaterThanOrEqual` and `LessThan` raise at 0001-01-01. Timestamp columns 
raise the same way at `9999-12-31 23:59:59.999999` and `0001-01-01 00:00:00`. 
`EqualTo` and `In` work. `day`, `hour`, `identity` and `bucket` partitions 
return the right rows. 9999-12-31 is a common "no end date" value.
   
   **Cause.** The projections step one day past the bound. The strict 
projection turns `<= 9999-12-31` into `< 10000-01-01` 
([transforms.py#L1066-L1070](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/transforms.py#L1066-L1070)).
 The inclusive projection turns `> 9999-12-31` into `>= 10000-01-01` 
([transforms.py#L1041-L1046](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/transforms.py#L1041-L1046)).
 Both then convert the day count to a Python `date` 
([datetime.py#L242-L243](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/utils/datetime.py#L242-L243)),
 and Python dates stop at year 9999. Java computes years and months with 
`LocalDate` 
([DateTimeUtil.java#L207-L225](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java#L207-L225)),
 which has a much wider r
 ange.
   
   #### 2. Filters on `inf` in a `float` column return wrong rows
   
   `EqualTo` and `In` miss the `inf` row, and `NotEqualTo` keeps it. 
`LessThanOrEqual(x, -math.inf)` misses the `-inf` row. Every layout behaves the 
same way. `double` columns are not affected.
   
   **Cause.** A Python float becomes a `DoubleLiteral`. 
[`DoubleLiteral.to(FloatType)`](https://github.com/apache/iceberg-python/blob/33b4ce511ec0c9dc0f74e38fff5d478ef2ff47dd/pyiceberg/expressions/literals.py#L434-L440)
 returns `FloatAboveMax` for any value above the float32 maximum, and `inf` is 
above it. Binding then turns `EqualTo` into `AlwaysFalse` and `NotEqualTo` into 
`AlwaysTrue`. Java has the same conversion 
([Literals.java#L372-L380](https://github.com/apache/iceberg/blob/2517ec2a7f70721166bd2b2ca0061984e4eb2bdc/api/src/main/java/org/apache/iceberg/expressions/Literals.java#L372-L380)).
 Java engines such as Spark pass a `Float` for a float column, so they do not 
take this path. In PyIceberg, every Python float does. `inf` and `-inf` are 
valid float values, so they could convert to `FloatLiteral`.
   
   #### Already reported
   
   While probing, I also hit three known issues. I do not repeat them here.
   
   - #3498 item 3, the `TypeError` for a null identity partition under an `Or` 
filter, still reproduces on `main`. The stale bot closed its fix, #3817.
   - apache/arrow#51491: Parquet row-group statistics drop NaN rows for `!=` 
and `is_nan`.
   - apache/arrow#46183: `isin([0.0])` drops a row group whose values are all 
`0.0`.
   
   ### How I found them
   
   I used [Temper](https://github.com/turbolytics/temper) to audit each table 
layout against the same scan guarantees. Every layout holds the same rows. The 
reference is the same filter written as plain Python over the appended rows. 
Temper helped identify these bugs. The table shows the result.
   
   | Guarantee | unpartitioned | many files | identity | bucket | truncate | 
year | month | day | hour |
   |---|---|---|---|---|---|---|---|---|---|
   | random filters match Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
   | int and long extremes are found | ✅ | ✅ | ✅ | ✅ | ❌ #4403 | ✅ | ✅ | ✅ | ✅ |
   | date and time sentinels scan | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ 1 | ❌ 1 | ✅ | ✅ |
   | float `inf` is found | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 | ❌ 2 
|
   
   ✅ passes. ❌ fails, marked with the bug number. The random row runs 300 
seeded filters on each layout, and all of them matched.
   
   The map is 
[`MATRIX.md`](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/MATRIX.md),
 built from [these 
tests](https://github.com/turbolytics/iceberg-python/blob/6de93440a50ea37c96e204a321814db385385911/tests/table/test_temper_scan_layout.py)
 and [these 
declarations](https://github.com/turbolytics/iceberg-python/tree/6de93440a50ea37c96e204a321814db385385911/temper).
   
   I can send a PR for each bug, with regression tests in the existing test 
files. I'm happy to split them into separate issues.
   
   I used an AI assistant to run the tests and draft this issue. I verified 
each example on 0.12.0 and on `main`, with pyarrow 26.0.1 and Python 3.12.9.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to