Rodrigo-Palma opened a new pull request, #3997:
URL: https://github.com/apache/iceberg-python/pull/3997
Closes #3996.
`to_bytes` for `DecimalType` compared `abs(exponent)` against the type's
scale. A
`Decimal` carries the negated scale as its exponent, so a value with a
*negative* scale
passed the check as if it had the matching positive one.
`decimal_to_unscaled` then uses
only the digits, so the exponent is dropped and the value is written four
orders of
magnitude off, with no error.
```python
t = DecimalType(10, 2)
from_bytes(t, to_bytes(t, Decimal("1E+2"))) # 0.01, expected 100.00
```
`Decimal("100").normalize()` is exactly `Decimal("1E+2")`, so a normalized
value, or one
that comes out of arithmetic that trims trailing zeros, hits this.
It matters beyond the round trip: `to_bytes` writes the `lower_bounds` and
`upper_bounds`
of a data file (`_write_data_file_statistics` in `pyiceberg/manifest.py`)
and is used again
in `pyiceberg/io/pyarrow.py`. A bound written as `0.01` instead of `100.00`
makes scan
planning prune files that do hold matching rows, so a query returns fewer
rows without
reporting anything.
## Change
Compare the signed scale, `-exponent`, so a mismatching value raises the
`ValueError` the
function already raises for any other scale mismatch. Rescaling to the
type's scale was the
other option, but that widens the contract of a function that today requires
an exact
match, so this keeps the existing one.
Values with a positive scale are unaffected, and the error message is
unchanged for them:
`Decimal("123.4567")` against `decimal(7, 3)` still reports `: 4`.
## Tests
Two cases added to the existing `test_datetime_obj_to_bytes` parametrization
in
`tests/test_conversions.py`, one using the literal `Decimal("1E+2")` and one
using
`Decimal("100").normalize()`. Both fail without the source change (`DID NOT
RAISE`).
`make lint` and `make test` green locally (4188 passed, 5 skipped).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
