Also, if you don’t care about display, just set the *precision* to LC max. Numberformat is as others have posted, really a display function. After you are done, set the numberFormat back to default or “0”. I bit myself in the butt because I had a counter that I was using to append a number to a variable name, and it hosed my names.
Bob S On Jun 16, 2014, at 11:13 , <dfepst...@comcast.net> <dfepst...@comcast.net> wrote: > > > I am trying to set the numberFormat so that calculation is precise enough for > the situation. Since I don't know ahead of time how many decimal places will > be used, I wrote a function that I hoped would adjust things as necessary. > But when this function is called with parameters m = 1.09131 and n = .0000001 > and k = 1, it returns 1.09131 rather than 1.0913101. > The last couple of lines were added for testing, and the debugger shows that > the numberFormat is being set correctly but that the truncated value is put > into hold. > Can anybody see what is going wrong? > > > > Many thanks. > > > > David Epstein > > > > function preciseEnough m,n,k > -- return the value m + k*n > -- default numberFormat shows up to 6 decimal places > -- If m, n, or k has more than that precision, this function sets the > numberFormat with a margin of safety > -- before returning the answer > put length(m) - offset(".",m) into aPlaces > put length(n) - offset(".",n) into bPlaces > put length(k) - offset(".",k) into cPlaces > put the numberFormat into myString > if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then > get myString & "####" > set the numberFormat to it > end if > put the numberFormat into nf > put m + k*n into hold > return hold > end preciseEnough > _______________________________________________ > use-livecode mailing list > email@example.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode _______________________________________________ use-livecode mailing list firstname.lastname@example.org Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode