At 18:38 05/02/2009 -0600, Ely Schoenfeld wrote:
The precision error didn't happen when I added 2.34 plus -2.73, it only happened when adding 18282.34 plus -18282.73 so it must have to be related with the "size" of the operated numbers.
The numbers will be stored as "floating-point" quantities, so they will have a fixed number of significant figures (*not* fractional places). There appear to be about fifteen significant decimal digits, which corresponds to about fifty significant binary digits, so I'm guessing that the so-called mantissa is stored in six bytes. And so yes: the larger the number, the sooner along the fractional part any rounding error will appear. In fact, you have to go up only to 4.34 and -4.73 to see the effect appear.
Brian Barker --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
