rahulsmahadev opened a new pull request, #4434:
URL: https://github.com/apache/iceberg-python/pull/4434
# Rationale for this change
Bug: an identity partition on a `double` or `float` column writes each NaN
row into every NaN partition file when the NaNs differ in the sign bit.
`float("nan")` has the sign bit clear and `0.0 * float("inf")` has it set on
x86. `_determine_partitions` groups by the raw bits, so these are two
partitions. Each one selects its rows with `is_nan`, which matches every NaN
row:
```python
# table partitioned by identity(value), value is a double
tbl.append(pa.table({"id": [1, 2, 3], "value": [float("nan"), 0.0 *
float("inf"), 1.5]}))
sorted(tbl.scan().to_arrow()["id"].to_pylist()) # [1, 1, 2, 2, 3], expected
[1, 2, 3]
```
Fix: canonicalize NaN in the partition helper column before the group by, so
every NaN is one partition value, as the spec defines partition equality (Scan
Planning). The data column keeps its bits.
Java: Spark computes each row's `PartitionKey`, then `FanoutWriter` and
`ClusteredWriter` use `Comparators.forType` to group all NaNs together. This PR
gives `_determine_partitions`, shared by append and rewrites, the same grouping.
Remaining differences from Java:
- `-0.0` and `0.0` have the same duplication. #4274 fixes that in
`_partition_value_filter`, and this PR leaves that function alone.
- The partition value written is the canonical NaN. Java keeps the NaN of
the first row. Both are equal under the spec.
## Are these changes tested?
Yes, added `test_determine_partitions_identity_nan_sign_bits` (double and
float, `tests/io/test_pyarrow.py`), and the existing
`test_append_nan_to_identity_partitioned_table` (append and scan,
`tests/catalog/test_catalog_behaviors.py`) now has a `-NaN` value. Both fail on
main, where the scan returns ids 2 and 4 twice and `_determine_partitions`
returns two NaN partitions, and pass with this change.
## Are there any user-facing changes?
Yes, appends no longer write duplicate rows for NaN values with different
sign bits.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]