Hi Morten, On Thu, 2006-16-02 at 21:43 -0500, Morten Welinder wrote: > > It is actually not a bug. > > The problem is that there is no way to represent the exact value of > 12/5 with a finite number of base-2 digits. (In the same way you > cannot represent 1/3 in base 10.) Gnumeric then uses the closest > possible value instead of 12/5 and this happens to be a tiny bit less > than 2.4. (In the 1/3 case, you would write something like 0.33333 > which also happes to be slightly less than the value you aimed for.) > > The value that appears to be 22 and is computed as 10+2.4+...+2.4 > is thus a little bit less than 22 and the 22s therefore do not get > counted. > > Ok, so what can be done about it? > > 1. Stay with integers. It's actually possible to multiply all the > numbers by 5. It's not very intuitive, though. > > 2. Replace (max-min)/5 by (max-min)/5*(1+1e-15). That ensures that > the value used to approximate 12/5 is slightly bigger than 2.4
While I understand the explanation I am surprised that even if I ask for 25 decimal digits it doesn't show that it is slightly smaller than 22 but if I take its difference from 22 (the integer) it shows a small difference in the 7th or 8th digit. Andreas -- Andreas J. Guelzow Taliesin Software, Shelties, Pyr Sheps and Shetland Sheep
signature.asc
Description: This is a digitally signed message part
_______________________________________________ gnumeric-list mailing list [email protected] http://mail.gnome.org/mailman/listinfo/gnumeric-list
