I am trying to set the numberFormat so that calculation is precise enough for
the situation. Since I don't know ahead of time how many decimal places will
be used, I wrote a function that I hoped would adjust things as necessary.
But when this function is called with parameters m = 1.09131 and n = .0000001
and k = 1, it returns 1.09131 rather than 1.0913101.
The last couple of lines were added for testing, and the debugger shows that
the numberFormat is being set correctly but that the truncated value is put
into hold.
Can anybody see what is going wrong?
Many thanks.
David Epstein
function preciseEnough m,n,k
-- return the value m + k*n
-- default numberFormat shows up to 6 decimal places
-- If m, n, or k has more than that precision, this function sets the
numberFormat with a margin of safety
-- before returning the answer
put length(m) - offset(".",m) into aPlaces
put length(n) - offset(".",n) into bPlaces
put length(k) - offset(".",k) into cPlaces
put the numberFormat into myString
if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
get myString & "####"
set the numberFormat to it
end if
put the numberFormat into nf
put m + k*n into hold
return hold
end preciseEnough
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
