Chris for the fixing of the rounding I was thinking of taking the first 5 or 6 numbers after the decimal place and dropping the rest (like what Michael said). Would this lead to correct answers in majority of cases? (to the closest 5th or 6th decimal)
Generally, you'll want to leave the numbers untouched while Calculating stuff (i.e. don't truncate, etc) -- this will keep as much precision as possible. only the last bit will be rounded for most operations -- this of course will compound with multiplications/ divisions, and will accumulate with addition/subtraction, but there's no real alternative*. Usually, rounding is set such that the error will evenly round, so that it averages out to 0.
When Displaying the result, truncating is fine -- like Michael said, 11-15 digits, depending on the nature of the calculations involved.
(*) if you have the luxury of writing _all_ the processing code yourself, you can use GMP or some other arbitrary-precision library to have significantly more precision. This will be much slower, but won't have the same limitations that native floating point code will.
-- [ christopher wright ] [email protected] http://kineme.net/
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Do not post admin requests to the list. They will be ignored. Quartzcomposer-dev mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/quartzcomposer-dev/archive%40mail-archive.com This email sent to [email protected]
