On 7/28/06, Phil M <[EMAIL PROTECTED]> wrote:
On Jul 27, 2006, at 11:21 PM, Norman Palardy wrote:> On Jul 27, 2006, at 7:58 PM, William Squires wrote: > >> I get exactly 27 on Calculator.App... :O > > I dont When you refactor the expression, I also get 27 no matter what the precision is set to. But if I go back to the original formula ((72 / 2000) * 750) then I get the following value with Calculator precision set to 16: 26.9999999999999964 _______________________________________________ Unsubscribe or switch delivery mode: <http://www.realsoftware.com/support/listmanager/> Search the archives of this list here: <http://support.realsoftware.com/listarchives/lists.html>
Interesting discussion. To muddy the waters further, using Mathematica: N[72/2000, 100] (ie calculate to 100 significant digits of precision) you get: 0.03600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 No argument here, 72/2000 is really 0.036 exactly. As people said, this real number may not be representable exactly as a binary floating point number, however: N[72/2000] (calculate to machine precision) gives 0.036, implying that the machine is indeed able to represent 0.036 exactly. P. -- ------------------------------------------------------------------------------- Peter K. Stys, MD Professor of Medicine(Neurology), Senior Scientist Ottawa Health Research Institute, Div. of Neuroscience Ottawa Hospital / University of Ottawa Ontario, CANADA tel: (613)761-5444 fax: (613)761-5330 http://www.ohri.ca/profiles/stys.asp ------------------------------------------------------------------------------- _______________________________________________ Unsubscribe or switch delivery mode: <http://www.realsoftware.com/support/listmanager/> Search the archives of this list here: <http://support.realsoftware.com/listarchives/lists.html>
