https://bugs.documentfoundation.org/show_bug.cgi?id=130725
Bug ID: 130725
Summary: ImpSvNumberInputScan::StringToDouble may produce
inaccurate result
Product: LibreOffice
Version: unspecified
Hardware: All
OS: All
Status: UNCONFIRMED
Severity: normal
Priority: medium
Component: Calc
Assignee: [email protected]
Reporter: [email protected]
CC: [email protected]
In Calc, put "0.0042" to A1, "0.0043" to A2, "=A2-A1" to A3; make sure to see
19 decimal places in A3.
Expected result:
> 0.0001000000000000000
Actual result:
> 0.0000999999999999994
The problem here is the entered string in A1 has been converted to double
0.0042000000000000006, which is not the nearest double-precision value for
decimal "0.0042": 0.0041999999999999997 is. The conversion of the entered
string to double happens in ImpSvNumberInputScan::StringToDouble, which handles
integer and fractional parts of the string separately, and then multiplies the
fractional part by a power 10 exponent. As shown, this does not guarantee the
closest possible representation, resulting in immediately observable
inaccuracies in following calculations.
A robust conversion is needed here.
--
You are receiving this mail because:
You are the assignee for the bug._______________________________________________
Libreoffice-bugs mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
