Shahzeb-DataEngineer opened a new issue, #4441:
URL: https://github.com/apache/iceberg-python/issues/4441
### What happens?
`FloatLiteral.to(DecimalType)` and `DoubleLiteral.to(DecimalType)` convert
the float with `Decimal(self.value)`, which uses the exact binary expansion of
the double. Values whose decimal spelling ends in 5 then round the wrong way
under `ROUND_HALF_UP`:
```python
from pyiceberg.expressions.literals import literal
from pyiceberg.types import DecimalType
literal(2.675).to(DecimalType(10, 2)) # 2.67, expected 2.68
literal(1.005).to(DecimalType(10, 2)) # 1.00, expected 1.01
literal(-2.675).to(DecimalType(10, 2)) # -2.67, expected -2.68
```
The wrong value leaks into predicate binding, so a filter on a decimal
column uses the wrong literal:
```python
from pyiceberg.expressions import EqualTo
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, DecimalType
schema = Schema(NestedField(1, "price", DecimalType(10, 2), required=True))
EqualTo("price", 2.675).bind(schema, case_sensitive=True).literal # 2.67,
expected 2.68
```
### Expected behavior
Match Apache Iceberg Java, which converts with
`BigDecimal.valueOf(value).setScale(scale, RoundingMode.HALF_UP)` in
`Literals.java` (all four numeric literal types). `BigDecimal.valueOf(double)`
uses `Double.toString`, i.e. the shortest decimal that round-trips to the
double, so `2.675` quantizes to `2.68`. Spark and Postgres agree (`2.675` casts
to `2.68`).
### Root cause
`pyiceberg/expressions/literals.py`, `FloatLiteral.to` and
`DoubleLiteral.to` for `DecimalType`:
```python
Decimal(self.value).quantize(Decimal((0, (1,), -type_var.scale)),
rounding=ROUND_HALF_UP)
```
`Decimal(2.675)` is `2.67499999999999982...`, which quantizes down to
`2.67`. Using `Decimal(str(self.value))` instead gives `Decimal("2.675")`,
which quantizes to `2.68`, matching `BigDecimal.valueOf`.
### Environment
pyiceberg 0.12.0, Python 3.12, Linux.
Happy to open the fix PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]