Hey Victor. Double has 2 times the amount of accuracy as Real does because its twice the size in memory.
This is because analog floating point numbers are being stored in a binary computer. Check out this oversimplified example :P if you only had 2 bits to store the number to the right of a decimal point, thats only 4 combinations so youd have: 0 0.25 0.5 0.75 so if you made a variable of that type and tried to set it to 0.14, it would have to set it to 0.25 since that was the closest match that it can store. if you had 3 bits, it doubles the number of combinations to 8 so you would have: 0 0.125 0.25 0.375 0.5 0.625 0.75 0.875 so now if you had a variable of that type and you tried to store 0.14, it would really store 0.125 since that was the closest match you can store. if you had 4 bits, the number of combinations go up to 16, with 5 bits it becomes 32 combinations and so on. the more bits you have, the more accurate the number you can store is, but it can never be infinitely accurate. Thats why double gives a more precise calculation while real gives an incorect one, cause double has twice the number of bits and squared the amount of accuracy as real does. ----- Original Message ----- From: "Victor Timmons" <[EMAIL PROTECTED]> To: "RBASE-L Mailing List" <[EMAIL PROTECTED]> Sent: Wednesday, June 23, 2004 11:08 AM Subject: [RBASE-L] - Rounding REAL vs DOUBLE > I am doing a very simple formula that does some rounding. > > R>SET VAR vUpdateAmount REAL = .01 > R>SET VAR vBookListPrice CURRENCY = 3.50 > R>SET VAR vBookListPrice2 DOUBLE = NULL > R>SET VAR vBookListPrice2 = .vBookListPrice > R>SET VAR vBookListPrice2 = ((.vBookListPrice2 * > .vUpdateAmount)+.vBookListPrice2) > R>SHO VAR vBookListPrice2 > 3.5399999921769 > R>SET VAR vBookListPrice2 = (ROUND(.vBookListPrice2,2)) > R>SHO VAR vBookListPrice2 > 3.53 > > If I change vUpdateAmount to a DOUBLE I get the correct results 3.54 > Why is that?? Isnt .01 = .01 wheather is a REAL or a DOUBLE?? Could > someone run this a let me know if they get the same results? > > R>SET VAR vUpdateAmount DOUBLE = .01 > R>SET VAR vBookListPrice CURRENCY = 3.50 > R>SET VAR vBookListPrice2 DOUBLE = NULL > R>SET VAR vBookListPrice2 = .vBookListPrice > R>SET VAR vBookListPrice2 = ((.vBookListPrice2 * > .vUpdateAmount)+.vBookListPrice2) > R>SHO VAR vBookListPrice2 > 3.535 > R>SET VAR vBookListPrice2 = (ROUND(.vBookListPrice2,2)) > R>SHO VAR vBookListPrice2 > 3.54 > > > > > Victor Timmons > Tiz's Door Sales, Inc > 425-258-2391 >
