karubian opened a new issue, #4323: URL: https://github.com/apache/iceberg-python/issues/4323
### Apache Iceberg version main (development) ### Please describe the bug 🐞 `PartitionFieldStats.update` takes the summary bounds with Python's `min` and `max`, which treat -0.0 and 0.0 as equal and keep whichever came first: https://github.com/apache/iceberg-python/blob/c8d67230cfdc4ce883a5166c1c2616e7403bde5f/pyiceberg/manifest.py#L727-L738. 0.12.0 has the same code. So for an identity partition on a float or double column holding both zeros, the bounds in the manifest list depend on entry order: ```python from pyiceberg.conversions import from_bytes from pyiceberg.manifest import construct_partition_summaries from pyiceberg.partitioning import PartitionField, PartitionSpec from pyiceberg.schema import Schema from pyiceberg.transforms import IdentityTransform from pyiceberg.typedef import Record from pyiceberg.types import DoubleType, NestedField schema = Schema(NestedField(1, "x", DoubleType(), required=False)) spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="x")) for values in ([0.0, -0.0], [-0.0, 0.0]): (summary,) = construct_partition_summaries(spec, schema, [Record(v) for v in values]) print(values, from_bytes(DoubleType(), summary.lower_bound), from_bytes(DoubleType(), summary.upper_bound)) ``` ``` [0.0, -0.0] 0.0 0.0 [-0.0, 0.0] -0.0 -0.0 ``` The spec rules out both: "If -0.0 is a value of the partition field, the `lower_bound` must not be +0.0, and if +0.0 is a value of the partition field, the `upper_bound` must not be -0.0." https://iceberg.apache.org/spec/#manifest-lists Iceberg Java's `PartitionSummary` compares with `Comparators.forType`, which is `Double.compareTo` for doubles and puts -0.0 below 0.0: - https://github.com/apache/iceberg/blob/b56b4bf9d6f3f4547fd3773e8fd78239869fad4b/core/src/main/java/org/apache/iceberg/PartitionSummary.java#L84-L99 - https://github.com/apache/iceberg/blob/b56b4bf9d6f3f4547fd3773e8fd78239869fad4b/api/src/main/java/org/apache/iceberg/types/Comparators.java#L40-L41 Both orders should give a lower bound of -0.0 and an upper bound of 0.0. #4274 fixes a different signed zero bug, rows written twice on append, and doesn't touch the summaries. ### Willingness to contribute - [X] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
