Date sent: Tue, 11 May 1999 12:05:30 -0700
From: Mark Nudelman <[EMAIL PROTECTED]>
> Mike Davis wrote:
> > 12345678.123 gets converted correctly to 12345678.123 and,
> > 12345678.1234 gets converted correctly to 12345678.1234 even,
> > 12345678.12345678 gets converted correctly to 12345678.12345678
> >
> > As soon as my IntegerPart exceeds 8 digits, the FracPart is incorrect
> > (rounded) regardless of how many digits are displayed.
> >
> > Bottom line is this has to do with rounding. But why does this only
> > happen when the IntegerPart exceeds 8 digits.
>
> I think the problem is not that the integer part exceeds 8 digits, but
> that the total number of digits (integer and fractional part) exceeds
> about 16 digits. A double can only hold a certain number of bits of
> precision (I think 52, as you pointed out), which works out to about 16
> decimal digits. It doesn't matter where the decimal point is within those
> 16 digits. You'll find (if I'm correct), that numbers like
> 1234567891234.56 will work ok, since there's a total of less than 16
> digits, even though the integer part is more than 8 digits. As for a
> workaround, there's not much you can do. If you need more than 16 digits
> of precision, you'll need to store your numbers in something other than a
> double, but you'll need to do a lot of work to write code to do arithmetic
> on your new type of number. --Mark
I have been giving this a lot of thought and I just think it's the
fact that not all decimals can be represented to 9 decimal places,
even within the 52 bits. As the size of the integer part increases
more and more of these 52 bits are used up to represent the integer
part. So for large numbers there are not enough remaining bits of
precision to represent the decimal part.
For instance: 12345678.12 gives 12345678.119999999 rounding error and
that is within 16 digits. The worst error occurs in the 9th decimal
spot. So, I think if I limit my NumberOfDecPlaces to 8 and the total
number of digits to 16, that my function rounding will make the
display look right.
--
-----------------------------------------------------------------
Discussion Group: http://www.halcyon.com/ipscone/wwwboard/
Protect your constitutional rights. Your favorite one may be next!
-----------------------------------------------------------------
