I am working on a Double To Ascii conversion routine. FlpAToF has
some limitations (like only exp to 99) that I wanted to overcome.
If you understand the inner workings of the double format and MathLib,
please read this carefully and maybe you can point me to the solution
to my problem.
Anyway, my routine works fairly well except for one strange thing.
I am having problems with one type of input. When I enter (with
output set to display 9 decimal digits):
123456789.123456789 it gets converted to 123456789.123456791 (slight
error).
This looks like a rounding problem because I am trying to display
more than 16 significant digits. But rounding should occur when
I convert my input to a DoubleVal, since 64 bits can only hold 2^52
for significand. But there is more to the problem.
I know the problem is NOT in my AsciiToDouble conversion routine.
I use, in part:
IntegerPart = trunc(DoubleVal); // MathLib function
FracPart = DoubleVal - IntegerPart;
I inserted one line just before the above code fragment:
DoubleVal = 123456789.123456789; // just for debug.
IntegerPart = trunc(DoubleVal);
FracPart = DoubleVal - IntegerPart;
but I get the same symptoms. So, that rules out my AsciiToDouble
conversion routine as the source of the problem.
Then, I inserted a debug statement after the FracPart calculation:
IntegerPart = trunc(DoubleVal);
FracPart = DoubleVal - IntegerPart;
FracPart = .123456789; // just for debug
When I entered 123456789.123456789, I get the correct result. So,
this means that something is going wrong in my FracPart calculation.
Now I think I COULD understand this given the 18 digits of double
which must produce a rounded DoubleVal. I also understand that a
decimal number can produce rounding. But what makes this a PUZZLE is
that this only occurs when the integer part exceeds 8 digits.
For instance:
12345678.123 gets converted correctly to 12345678.123 and,
12345678.1234 gets converted correctly to 12345678.1234 even,
12345678.12345678 gets converted correctly to 12345678.12345678
As soon as my IntegerPart exceeds 8 digits, the FracPart is incorrect
(rounded) regardless of how many digits are displayed.
Bottom line is this has to do with rounding. But why does this only
happen when the IntegerPart exceeds 8 digits.
I don't know if this is a MathLib problem; a double format problem
or what. I'm pretty sure it is not any of my conversion routines
because I have tested it with hard coded doubles and it works with
those. Anyone know why this happens? More importantly what is a
workaround? Different FracPart calculation?
--
-----------------------------------------------------------------
Discussion Group: http://www.halcyon.com/ipscone/wwwboard/
Protect your constitutional rights. Your favorite one may be next!
-----------------------------------------------------------------
