https://bugs.documentfoundation.org/show_bug.cgi?id=90997
--- Comment #2 from Dane Maslen <[email protected]> --- I agree that it's a precision issue, but not in the way you believe. I think your pointing out that rounding to 15 significant figures is another workaround to the problem allows me to conceptually identify what is going wrong in Calc. Irrespective of precision limits the contents of a given cell should always compare as being equal to the contents of that same cell. The formula =IF(A1=A1,1,0) does indeed always give the correct result, i.e. 1. Likely conclusion: Calc directly compares the contents of A1 to the contents of A1. Furthermore when A1 is =RAND() B1 is =A1 C1 is =IF(A1=B1,1,0) C1 always has the correct result, i.e. 1. Likely conclusion: Not only does Calc directly compare the contents of A1 to B1 but also the assignment of a value to B1 from A1 is either being done directly or using intermediate variables of precision at least as great as those used to store the cell values. The fact that COUNTIF(A1:A1,"="&A1) does not always give the correct answer indicates that the contents of the cell A1 are not being compared directly with the contents of each of the cells in the range A1 to A1 (as noted above, comparing A1 directly to A1 always works). My guess would be that the contents of the cell A1 are copied to an intermediate variable and that that is what is then compared to each of the cells in the range A1 to A1. The fact that COUNTIF(A1:A1,"="&A1) does give the correct answer when the contents of all the cells have been rounded to 15 significant figures gives a clue as to what is going wrong. It suggests that Calc stores cell values with a precision greater than 15 sf but is using an intermediate variable with precision of only 15 sf when evaluating the COUNTIF comparisons. That's a bug. I've been a programmer long enough to know that if one wants to compare two values that are known to high precision, one doesn't first assign one of those values to a variable of low precision. In summary: when evaluating COUNTIF Calc should be using an intermediate variable of precision at least equal to the precision with which it stores cell values. -- You are receiving this mail because: You are the assignee for the bug.
_______________________________________________ Libreoffice-bugs mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
