On Nov 23, 2006, at 5:47 PM, Lennox Jacob wrote:
Hello,
I have an app in which I perform some calculations.
The problem is that when I use large numbers like 60000.23 +
25000.23 + 25000.23 I get 110000.70 instead of 110000.69
Actually, Lennox, the result of that addition in Rb is
110000.69000000, which is accurate to 15 or 16 significant figures as
one would expect per the IEEE floating point standard for a double.
You are viewing a rounded result, rather than the full precision
result stored in memory.
Here is how you can view the result of that calculation correctly to
the precision of a double:
dim r as string = Format(60000.23 + 25000.23 + 25000.23,
"#.###########")
MsgBox r // shows 110000.69
You can see a decimal representation of the numerical result stored
in memory by adding a few more #'s past the decimal point above, like
this:
dim r as string = Format(60000.23 + 25000.23 + 25000.23,
""#.################"")
MsgBox r // shows 110000.6900000000023283
This result, 110000.6900000000023283, is the floating point number
which Rb is using to approximate the result of the calculation. As
you can see, it is exact to 17 significant digits in this case.
In other words, the 'problem' you are seeing is a result of the
manner in which the result is displayed. It doesn't reflect the
precision of the result as calculated and represented internally by Rb.
Best,
Jack
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>
Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>
