The numberformat doesn't affect precision. It only affects output as is shown by

on mouseUp
   set the numberformat to "00"
   put 0.01*1 into x
   set the numberformat to "00.00"
   put x*1
end mouseUp

It even only affects the current handler and doesn't affect precision in calling handlers:

on mouseUp
   // default is "0.######"
   put a1() into x
   put x*1
end mouseUp

function a1
   set the numberformat to "00"
   return 0.01*1
end a1

The above script returns 0.01 instead of 00.

I don't think LiveCode allows you to adjust the precision as other languages to with e.g. signed and unsigned integers and floating precision. You'd have to write special routines for this.

I think there is quite a lot of discussion to find in the archives about high precision and floating point numbers.

Best regards,

Mark Schonewille

Economy-x-Talk Consulting and Software Engineering
Homepage: http://economy-x-talk.com
Twitter: http://twitter.com/xtalkprogrammer
KvK: 50277553

Installer Maker for LiveCode:

Buy my new book "Programming LiveCode for the Real Beginner" http://qery.us/3fi

LiveCode on Facebook:

On 6/16/2014 20:13, dfepst...@comcast.net wrote:

I am trying to set the numberFormat so that calculation is precise enough for 
the situation.  Since I don't know ahead of time how many decimal places will 
be used, I wrote a function that I hoped would adjust things as necessary.
But when this function is called with parameters m = 1.09131 and n = .0000001 
and k = 1, it returns 1.09131 rather than 1.0913101.
The last couple of lines were added for testing, and the debugger shows that 
the numberFormat is being set correctly but that the truncated value is put 
into hold.
Can anybody see what is going wrong?

Many thanks.

David Epstein

function preciseEnough m,n,k
    -- return the value m + k*n
    -- default numberFormat shows up to 6 decimal places
    -- If m, n, or k  has more than that precision, this function sets the 
numberFormat with a margin of safety
    -- before returning the answer
    put length(m) - offset(".",m) into aPlaces
    put length(n) - offset(".",n) into bPlaces
    put length(k) - offset(".",k) into cPlaces
    put the numberFormat into myString
    if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
       get myString & "####"
       set the numberFormat to it
    end if
    put the numberFormat into nf
    put m + k*n into hold
    return hold
end preciseEnough

use-livecode mailing list
Please visit this url to subscribe, unsubscribe and manage your subscription 

Reply via email to