At 13:28 04/02/2009 -0600, Ely Schoenfeld wrote:
I discovered something odd with OOo 3.0.1 Calc.
[...]
If I add more decimals to the cell format, I get something like
-0.390000, that's ok.
But If I copy the last cell and then paste special with only text,
numbers and dates checked, I obtain:
-0.389999999999418
It's worth looking at exactly what is happening here. The three
fractions you quote, 0.34, 0.73, and 0.39 all terminate as decimal
fractions, of course. They are 34/100, 73/100, and 39/100, of course
- and they terminate because the denominator, 100, contains only
prime factors that appear in the number base, 10. But, expressed as
binary fractions, these do not terminate. In fact, they all finish
with a twenty-digit recurring pattern. (I think this is because of
the two factors of 5 buried in that 100 denominator: the twentieth
power of two is the lowest one which is one more than a multiple of
25.) So any representation of such fractions in the binary workings
of a computer are almost bound to be rounded and not precise, no
matter how many binary digits are preserved.
There is, I think, a direct solution to your needs. Go to Tools |
Options... | OpenOffice.org Calc | Calculate, and tick the "Precision
as shown" option. If you have this in place when you perform the
copy-and-paste, you should see a difference. (It's the copy that
matters, in fact, not the paste.)
I trust this helps.
Brian Barker
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
