Thanks for everyones help. Chris for the fixing of the rounding I was thinking of taking the first 5 or 6 numbers after the decimal place and dropping the rest (like what Michael said). Would this lead to correct answers in majority of cases? (to the closest 5th or 6th decimal)
On Thu, Sep 24, 2009 at 10:24 PM, Christopher Wright <[email protected]>wrote: > A quick question here, why does 20/200.0 give me 0.10000000000000001 in >> cocoa >> I would of expected some weird rounding errors in Javascript, but I >> thought cocoa would be able to handle the above equation. >> > > Like Michael said, it's due to precision (Cocoa/QC use doubles, which are > 64bit floating point numbers). Javascript likely has the same precision (or > possibly half as much precision? Not sure if it's float or double-backed). > > If it can't help it, will it always round up, ie be 0.00000000000000001 >> above the correct answer. If so, i can work around it. >> > > > Rounding is a tricky subject, and you really _shouldn't_ depend on it being > a constant amount off because doubles/floats use relative precision (an > exponent and a magnitude) -- you'll likely just be adding more imprecision > -- doubles can handle 0.00000000000000001 without any problems, but if you > add that to 100000000, the result is still 100000000 because the small > amount is rounded away completely. However, if you add that to > 0.00000000000000001, you get 0.00000000000000002, which is twice as much as > you want. 0.00000000000000020/200 would have a rounded error much much much > smaller than the "standard" 0.00000000000000001 you've calculated. > > (0.00000000000000020/200 gives 1.00000000000000007154e-18, in this case -- > the ...7154 being the error) > > -- > [ christopher wright ] > [email protected] > http://kineme.net/ > >
_______________________________________________ Do not post admin requests to the list. They will be ignored. Quartzcomposer-dev mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/quartzcomposer-dev/archive%40mail-archive.com This email sent to [email protected]
