Sam Ruby wrote:
> I find it easier to talk about real examples than abstractions. I've
> done the following quickly, so forgive me if I get some detail wrong.
>
> A binary floating point number has 52 bits of fraction, and by
> assuming an implicit leading one, they get an additional bit. This
> means that 1.1 is stored as (for brevity, I'll use hex)
>
> [1].1999999999999
>
> A conversion of that to decimal128 would be equivalent to computing
>
> 4953959590107545m / 4503599627370496m
>
> Which would produce
>
> 1.099999999999999866773237044981215
That's incorrect. The correct answer is 1.100000000000000088817841970012523,
which is closer mathematically to 1.1.
> Repeating that for 1.2 produces
>
> 0x13333333333333L
> 5404319552844595m / 4503599627370496m
> 1.199999999999999955591079014993738
This one is correct.
Waldemar
_______________________________________________
Es-discuss mailing list
[email protected]
https://mail.mozilla.org/listinfo/es-discuss
