https://bugs.documentfoundation.org/show_bug.cgi?id=146604
Mike Kaganski <mikekagan...@hotmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|--- |NOTABUG
CC| |er...@redhat.com
Status|UNCONFIRMED |RESOLVED
--- Comment #1 from Mike Kaganski <mikekagan...@hotmail.com> ---
These are actually two slightly different issues (but both are related to the
same limit of precision of IEEE 754 64-bit floating-point numbers used in Calc
[1]).
> #1
> 1. enter the number 5,629,499,534,213.13 in cell A1
> 2. double the number in A2 with =2*A1
>
> 11,258,999,068,426.30
This is actually an artifact of *internal limit of displayed digits*. No matter
how many digits you define to show, Calc will not show more than 15 significant
digits - rounding to 15th digit; and will append zeros to the right if needed.
So it shows you 15 digits, which needs to round ".26" to ".3".
> #2
> 1. enter the number 4,503,599,627,370,496 in cell A1
> 2. enter the same number 4,503,599,627,370,496 in cell A2
> 3. in A3 sum the two above with =SUM(A1:A2)
>
> 9,007,199,254,740,990
This is the result of special handling of *integers*. Here the sum of two
integer numbers, each correctly representable in double, is equal or greater
than 2^53, which means it's no more correctly representable as whole number in
double (in fact, 2^53 is representable, but is already sharing the "slot" with
the next whole number). Note that the initial integer in case #2 is already 16
digits, not 15 as you write in the preamble.
Calc is *not* a proper tool for precise calculations with numbers having such a
precision. This is not a bug, but an inherent limitation: that is, it's a use
case which the application is *not* suitable for.
Note that Gnumeric (which does not try to round/limit shown numbers) would
create impression of "correct" calculations in these cases. But e.g. if in case
#2 you also add 1 to the result, you will see the problem. It's just a
different way to display the inherent limited precision problem, but not a fix.
Other spreadsheet applications (Excel, Google Sheets) behave the same as Calc
(or worse - Excel doesn't accept 16-digit whole numbers in the first place, so
entering 4503599627370496 produces 4503599627370490 in the cell).
[1] https://wiki.documentfoundation.org/Faq/Calc/Accuracy
--
You are receiving this mail because:
You are the assignee for the bug.
