https://bugs.freedesktop.org/show_bug.cgi?id=86005
--- Comment #3 from Jindrich Svorc <[email protected]> --- (In reply to Robinson Tryon (qubit) from comment #1) > (In reply to Jindrich Svorc from comment #0) > > Created attachment 109088 [details] > > Spreadsheet sample and a screenshot of my laptop just in case it works on > > your computer > > > > Problem description: > > In some cases value 1 is not correctly calculated. > > > > Please see attached files for explanation. I am attaching ods file as well > > as a picture. The reason is that I tried it in MS Excel and it shows the > > same problem as well. So I am not sure whether the problem is not somewhere > > in windows or in my machine rather than in Calc. > > I tried it in OpenOffice 3.1.1 on RedHat 5 and it shows the same issue. > > I think this is just a matter of floating-point arithmetic. > > > Current behavior: > > equation > > =INT(8.45)+(8.45-INT(8.45))/0.6-7.75 gives 0.99999999999999800000 > > the same for Floor function > > =FLOOR(8.45,1)+(8.45-FLOOR(8.45,1))/0.6-7.75 gives 0.99999999999999800000 > > Note that there are 14 9's and 15 non-zero places in the number: > 0.999 999 999 999 998 000 00 > > If we compare that result with a simpler calculation such as =1/3 we find a > similar situation with only 15 decimal places displaying the expected result > values: > 0.333 333 333 333 333 000 00 I agree with 1/3 but 8+(8.45-8)/0.6-7.75 should give exact 1, not 0.9999999999... 1/3 is periodically ended number but the result of the equation above is not. If you calculate it by hand you will get 1, not something close to 1. That's what puzzles me. Interestingly, if you change the equation to get result = 2 it calculates it correctly. I would not care too much since I usually don't need the 15th place but the problem is that INT function gives me 0 instead of 1 so it suddenly creates a big difference. > > So it appears that while LibreOffice may format a number to 20 places, the > internal calculations are only valid up to 15 places. -- You are receiving this mail because: You are the assignee for the bug.
_______________________________________________ Libreoffice-bugs mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
