https://bugs.documentfoundation.org/show_bug.cgi?id=146604
Mike Kaganski <mikekagan...@hotmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |NOTABUG CC| |er...@redhat.com Status|UNCONFIRMED |RESOLVED --- Comment #1 from Mike Kaganski <mikekagan...@hotmail.com> --- These are actually two slightly different issues (but both are related to the same limit of precision of IEEE 754 64-bit floating-point numbers used in Calc [1]). > #1 > 1. enter the number 5,629,499,534,213.13 in cell A1 > 2. double the number in A2 with =2*A1 > > 11,258,999,068,426.30 This is actually an artifact of *internal limit of displayed digits*. No matter how many digits you define to show, Calc will not show more than 15 significant digits - rounding to 15th digit; and will append zeros to the right if needed. So it shows you 15 digits, which needs to round ".26" to ".3". > #2 > 1. enter the number 4,503,599,627,370,496 in cell A1 > 2. enter the same number 4,503,599,627,370,496 in cell A2 > 3. in A3 sum the two above with =SUM(A1:A2) > > 9,007,199,254,740,990 This is the result of special handling of *integers*. Here the sum of two integer numbers, each correctly representable in double, is equal or greater than 2^53, which means it's no more correctly representable as whole number in double (in fact, 2^53 is representable, but is already sharing the "slot" with the next whole number). Note that the initial integer in case #2 is already 16 digits, not 15 as you write in the preamble. Calc is *not* a proper tool for precise calculations with numbers having such a precision. This is not a bug, but an inherent limitation: that is, it's a use case which the application is *not* suitable for. Note that Gnumeric (which does not try to round/limit shown numbers) would create impression of "correct" calculations in these cases. But e.g. if in case #2 you also add 1 to the result, you will see the problem. It's just a different way to display the inherent limited precision problem, but not a fix. Other spreadsheet applications (Excel, Google Sheets) behave the same as Calc (or worse - Excel doesn't accept 16-digit whole numbers in the first place, so entering 4503599627370496 produces 4503599627370490 in the cell). [1] https://wiki.documentfoundation.org/Faq/Calc/Accuracy -- You are receiving this mail because: You are the assignee for the bug.