On Jul 27, 2006, at 9:58 PM, William Squires wrote:
??!?
I get exactly 27 on Calculator.App... :O
Even if I refactor as (72 * 2000) / 750, I still get 27 exactly.
Which makes sense as - in base 10 - dividing by 2000 will produce a
rational, non-repeating decimal (unless you start with an
irrational number in the numerator - like pi, say, or sqrt(2) - or
the numerator is itself a repeating decimal fraction like 1/3.)
Calculator has a "precision" function, which will round to the range
selected. Set the precision to "2" and enter "2 / 3" which you know
should be 0.6666666666666666666666666 but is displayed as "0.67".
Calculator is also taking a numeric value and displaying it on the
screen as an string (using a precision function).
The Str() function is similar in that it also provides some sort of
precision rounding, but often as not, it will have undesirable
results (scientific notation). This is why the Format() function is
more often used, but this function does *not* have a precision
rounding feature (although it will trim to the format you specify).
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>
Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>
