rahulsmahadev opened a new pull request, #4434:
URL: https://github.com/apache/iceberg-python/pull/4434

   # Rationale for this change
   
   Bug: an identity partition on a `double` or `float` column writes each NaN 
row into every NaN partition file when the NaNs differ in the sign bit. 
`float("nan")` has the sign bit clear and `0.0 * float("inf")` has it set on 
x86. `_determine_partitions` groups by the raw bits, so these are two 
partitions. Each one selects its rows with `is_nan`, which matches every NaN 
row:
   
   ```python
   # table partitioned by identity(value), value is a double
   tbl.append(pa.table({"id": [1, 2, 3], "value": [float("nan"), 0.0 * 
float("inf"), 1.5]}))
   sorted(tbl.scan().to_arrow()["id"].to_pylist())  # [1, 1, 2, 2, 3], expected 
[1, 2, 3]
   ```
   
   Fix: canonicalize NaN in the partition helper column before the group by, so 
every NaN is one partition value, as the spec defines partition equality (Scan 
Planning). The data column keeps its bits.
   
   Java: Spark computes each row's `PartitionKey`, then `FanoutWriter` and 
`ClusteredWriter` use `Comparators.forType` to group all NaNs together. This PR 
gives `_determine_partitions`, shared by append and rewrites, the same grouping.
   
   Remaining differences from Java:
   - `-0.0` and `0.0` have the same duplication. #4274 fixes that in 
`_partition_value_filter`, and this PR leaves that function alone.
   - The partition value written is the canonical NaN. Java keeps the NaN of 
the first row. Both are equal under the spec.
   
   ## Are these changes tested?
   
   Yes, added `test_determine_partitions_identity_nan_sign_bits` (double and 
float, `tests/io/test_pyarrow.py`), and the existing 
`test_append_nan_to_identity_partitioned_table` (append and scan, 
`tests/catalog/test_catalog_behaviors.py`) now has a `-NaN` value. Both fail on 
main, where the scan returns ids 2 and 4 twice and `_determine_partitions` 
returns two NaN partitions, and pass with this change.
   
   ## Are there any user-facing changes?
   
   Yes, appends no longer write duplicate rows for NaN values with different 
sign bits.
   
   Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to