Rodrigo-Palma opened a new issue, #3996:
URL: https://github.com/apache/iceberg-python/issues/3996
`to_bytes` for `DecimalType` takes the absolute value of the exponent before
comparing it
against the type's scale:
```python
_, digits, exponent = value.as_tuple()
exponent = abs(int(exponent))
if exponent != primitive_type.scale:
raise ValueError(...)
```
A `Decimal` carries the negated scale as its exponent, so a value with a
*negative* scale
passes this check as if it had the matching positive one.
`decimal_to_unscaled` then uses
only the digits, and the exponent is dropped:
```python
sign, digits, _ = value.as_tuple()
return int(Decimal((sign, digits, 0)).to_integral_value())
```
The value is written four orders of magnitude off, with no error.
## Reproduction
```python
from decimal import Decimal
from pyiceberg.conversions import to_bytes, from_bytes
from pyiceberg.types import DecimalType
t = DecimalType(10, 2)
print(from_bytes(t, to_bytes(t, Decimal("1E+2")))) # 0.01, expected 100.00
print(from_bytes(t, to_bytes(t, Decimal("5E+1")))) # 0.50 for decimal(10,
1) -> 0.5, expected 50.0
```
This is not an exotic input. `Decimal.normalize()` produces exactly this
form:
```python
Decimal("100").normalize() # Decimal('1E+2')
```
so a value that has been normalized, or that comes out of arithmetic that
trims trailing
zeros, hits it.
## Impact
`to_bytes` writes the `lower_bounds` and `upper_bounds` of a data file
(`pyiceberg/manifest.py`, `_write_data_file_statistics`), and the same
conversion is used
in `pyiceberg/io/pyarrow.py`. A bound written as `0.01` instead of `100.00`
makes scan
planning prune files that do hold matching rows, so a query silently returns
fewer rows
than it should.
Values with a positive scale are unaffected: `Decimal("100.00")` round-trips
correctly.
## Suggested fix
Compare the signed scale, `-exponent`, against `primitive_type.scale`, so a
mismatching
value is rejected instead of being silently rescaled. Rescaling the value to
the type's
scale would be the other option, but that widens the contract of a function
that today
requires an exact match.
Happy to open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
