akupchinskiy opened a new issue, #12048:
URL: https://github.com/apache/arrow-rs/issues/12048
## Describe the bug
`cast` from Float64/Float32 to a decimal, `float_to_decimal` and
`float_to_decimal_opt` scale in
floating point: `float_to_decimal_checked` computes
`D::Native::from_f64((10^scale * x).round())`.
Once `10^scale * x` is 2^53 or more, an `f64` no longer holds every integer.
The product is then
off by up to half its ulp, and the low digits of the result are noise:
| input | target | result | expected |
|---|---|---|---|
| `295161.0` | `Decimal128(31, 15)` | `295160.999999999967232` |
`295161.000000000000000` |
| `1000.0` | `Decimal128(38, 20)` | `999.99999999999991611392` |
`1000.00000000000000000000` |
| `123456789.0` | `Decimal128(38, 20)` | `123456788.99999999188514373632` |
`123456789.00000000000000000000` |
| `100000.25` | `Decimal128(38, 17)` | `100000.24999999999836160` |
`100000.25000000000000000` |
| `3.14159` | `Decimal128(38, 22)` | `3.1415899999999998558208` |
`3.1415900000000000000000` |
These results are neither the exact binary value of the input (`295161.0` is
exactly 295161) nor
its decimal representation. Some inputs come out right by chance (`1234.56`
or `0.1` at scale 20).
The share of wrong results, measured against the input's shortest decimal
representation:
| inputs | scale 9 | 12 | 15 | 18 | 20 |
|---|---|---|---|---|---|
| integers 1 to 10^9 | 0% | 87.5% | 99.8% | 100% | 100% |
| amounts with 2 decimals, up to 10^6 | 0% | 5.1% | 80.8% | 99.6% | 100% |
| random doubles in [0, 1000) | 0% | 0.8% | 92.5% | 99.9% | 100% |
One downstream effect: DataFusion compares `DECIMAL` with `DOUBLE` by
casting the double to
`Decimal128(30, 15)` (apache/datafusion#14272). Reconciling 100,000
identical amounts,
`DECIMAL(18,2)` against `DOUBLE`, finds only 67,057 matches and reports
16,471 as "underpaid".
#10508 fixed the reverse direction (decimal to float not correctly rounded)
in October 2026.
## To Reproduce
```rust
use arrow_array::types::Decimal128Type;
use arrow_cast::cast::float_to_decimal;
assert_eq!(
float_to_decimal::<Decimal128Type>(295161.0, 31, 15).unwrap(),
295161 * 10_i128.pow(15)
); // fails: 295160999999999967232
```
## Expected behavior
`295161.0` at scale 15 is `295161 * 10^15`. When the scaled value reaches
2^53, the cast should
stop scaling in floating point and scale the number exactly. The digits to
scale are the shortest
decimal representation of the input, i.e. the digits arrow displays it with
(`ryu`). This is also
what PostgreSQL and Spark (`BigDecimal(Double.toString)`) use. Below 2^53
the current results can
stay unchanged. I have a PR that does this and makes the common
float-to-decimal casts faster.
## Additional context
Two points maintainers may want to decide separately. Neither is changed by
the PR.
1. **Rounding next to a boundary, below 2^53.** The current rounding of the
floating-point product
differs from PostgreSQL and Spark there:
| value, scale | arrow | PostgreSQL / Spark | why |
|---|---|---|---|
| `1.005`, scale 2 | `1.00` | `1.01` | `1.005 * 100` is
`100.49999999999999` |
| `0.145`, scale 2 | `0.14` | `0.15` | |
The current result is defensible: the double nearest to 1.005 is
`1.00499999999999989…`. If
arrow wants the PostgreSQL / Spark semantics everywhere, the same exact
path could be used
near rounding boundaries too. That costs extra work for every value in
the hot loop.
2. **Float32 and Float16.** Arrays of these types are widened to `f64`
before the conversion.
At high scales the digits are therefore those of the widened value:
`0.1f32` becomes
`0.10000000149011612`, not `0.1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
