https://bugs.documentfoundation.org/show_bug.cgi?id=138920
Eike Rathke <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |NEW
CC| |[email protected]
Ever confirmed|0 |1
--- Comment #9 from Eike Rathke <[email protected]> ---
We're trying to round and due to the rounding even have some wrong display
strings resulting of the rounding to 15 decimals, but that 0.269985590000001
*is* already rounded; so yes, the entire display string conversion from binary
IEEE 754 to decimal needs to be reworked, but any "user expects" and "easy fix"
claim is just off.
Decimal input 9.25178559
nearest binary IEEE 754 representable value
9.251785590000000780719346948899328708648681640625
(1.001010000000111010100000101001110101010011101110001 * 2^3)
Decimal input 8.9818
nearest binary IEEE 754 representable value
8.98179999999999978399500832892954349517822265625
(1.00011111011010101110011111010101011001101100111101 * 2^3)
Subtraction:
1.0001010001110111000110100011110111000011110111000000 * 2^-2
0.269985590000000996724338619969785213470458984375
rounded to 15 decimals 0.269985590000001
Yes, unfortunately rounding to 15 decimals is not enough *in this* case, but 14
would be even more wrong in other cases, and sometimes 17 are required..
Fwiw, the nearest representable binary for 0.26998559 is
0.26998558999999999752361645732889883220195770263671875
(1.000101000111011100011010001111011100001111011010111 * 2^-2)
--
You are receiving this mail because:
You are the assignee for the bug._______________________________________________
Libreoffice-bugs mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
