On 6/16/2014, 1:13 PM, dfepst...@comcast.net wrote:


I am trying to set the numberFormat so that calculation is precise enough for 
the situation.  Since I don't know ahead of time how many decimal places will 
be used, I wrote a function that I hoped would adjust things as necessary.
But when this function is called with parameters m = 1.09131 and n = .0000001 
and k = 1, it returns 1.09131 rather than 1.0913101.
The last couple of lines were added for testing, and the debugger shows that 
the numberFormat is being set correctly but that the truncated value is put 
into hold.
Can anybody see what is going wrong?



Many thanks.



David Epstein



function preciseEnough m,n,k
    -- return the value m + k*n
    -- default numberFormat shows up to 6 decimal places
    -- If m, n, or k  has more than that precision, this function sets the 
numberFormat with a margin of safety
    -- before returning the answer
    put length(m) - offset(".",m) into aPlaces
    put length(n) - offset(".",n) into bPlaces
    put length(k) - offset(".",k) into cPlaces
    put the numberFormat into myString
    if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
       get myString & "####"
       set the numberFormat to it
    end if
    put the numberFormat into nf
    put m + k*n into hold
    return hold
end preciseEnough

As Mark said, numberformat is only retained during the local handler so you need to reset it each time the handler runs.

When you use #, it means to include a numerical value in that postion only if there is an actual value there; if there is no value then that position is ignored. If you use 0 in the numberformat instead, empty positions are padded with zeros. So, to get the precision you want, you don't need to calculate the number of places/positions, just use the # as you are now. Include enough #s to cover your longest anticipated number, up to LC limits (I think that's 16 places.)

Since numberformat only affects the display, you need to force the calculation from a numerical value to a text value. You can do that by simply putting the result of the calculation into a field. If you want it in a variable, you can use value() to do that instead.

So your whole handler can be like this:

function preciseEnough m,n,k
  put m + k*n into hold
  set the numberformat to "0.##############" -- add more if you want
  put value(hold) into hold
  return hold
end preciseEnough

--
Jacqueline Landman Gay         |     jac...@hyperactivesw.com
HyperActive Software           |     http://www.hyperactivesw.com

_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Reply via email to