Barry, Search the archives for anything on wide-zero for detailed explanation, but I'll have a go anyway.
It is all to do with binary representation of floating point numbers. If you convert an integer from decimal to binary the resultant value is always precise and unambiguous, the binary can always be converted back to decimal with the correct result. If you try this with a floating point number that isn't always true, a small percentage of decimal values between 0 and 1 do not have a precise binary representation, you can get close but you can't get there exactly. So, if you perform arithmetic operations on these binary representations you stand the chance of getting very small inaccuracies, or rounding errors. Many programming environments (including Unidata) will resolve these for you - I'm not sure how (help me Ken ?) but possibly by converting fp values to integers first, then calculating, then converting back Universe performs the calculation on the fp numbers themselves and uses a tunable parameter (WIDE-ZERO) to allow you to specify the point at which very small values can be considered equivalent to zero. The default value is an industry-standard (IEEE) value of 2.91E-11 which is roughly 1 / 2 pwr 35. By totalling values in external (OCONV) format in Universe you will occasionally fall foul of this and find that the total is "out" by a small amount - although 0.000005 is larger than I would expect, may depend on the number of values you are totalling. You can "tune" your system by tweaking the value of the wide-zero parameter (default 0x3dc00000) or, as recommended by many gurus on this list, you should always perform arithmetic on the internal (ICONV) values so if you modify your F55 definition to ICONV the value with an MD2 conversion then I think you'll find the total to be as expected ( but 100 times too large) Hth Piers -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Barry Rogen The output I get is quite bizarre. Instead of the expected value of 134878167.90 I get 134878167.890005 Any thoughts ??? Barry Rogen ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/
