On Tuesday 15 February 2005 10:53, Lourens Veen wrote: > Okay, so I've got a routine for calculating reciprocal mantissas that > uses two tables with 18-bit values, and returns a new 18 bit > mantissa. I have a maximum positive error of 1, and a maximum > negative error of 0. That is, the result is sometimes 1 too large and > sometimes spot on, but it's never too small. > > So, theoretically, that is 17 bits precision right?
Actually, talking "bits" of precision is not very precise. If a bit is incorrect (for some definition of correct) it might be very close to the true value or it might be very far. Also, a very small error can potentially carry to arbitrary higher bit positions. What I understand by "17 bit precision" is "17 bits are identical", but you can see from the above that this won't always hold in a useful way. Other possible interpretations are "absolute error is less than 2**-bits" or "error range is less than 2**-bits". By either definition, you've got 17 bit accuracy. It's more useful to talk about relative error: (approx - exact)/approx Once I have the positive and negative relative error, I take log2 of the difference and call that the "bits of precision", which is just a useful yardstick. It's the relative error that tells the story. Mind you, I haven't dabbled much in the minutae of precision since back in school 30 years ago so my recollection might be a little, ahem, imprecise. One more thing to check for is, if the correct result is an exact power of 2, including negative, you should always produce the exact result. This is because people rely on such things as 1/2 being exactly .5. In particular, 1/1 should always be 1. > But how do I > round this to always get the correct 17 bit mantissa? It seems to be > impossible: > > in ap out aptr aprd > 00 00 0 0 0 > 00 01 0 0 1 > 01 01 1 0 1 > 01 10 1 1 1 > 10 10 1 1 1 > 10 11 1 1 0 > 11 11 0 1 0 > 11 00 0 0 0 > > where in is the two least significant bits of the correct 18-bit > result, ap the two least significant bits of the approximated 18-bit > number, out the correct least significant bit of the 17-bit result, > aptr the result of truncating the approximation, and aprd the result > of rounding the approximation. > > Seems like both rounding and truncating only give the right answer in > three out of four cases. I presume the out column is the rounded version of in? > Am I missing something? Yes: trying to get the bits to match exactly leads to madness. How does the relative error look? Regards, Daniel _______________________________________________ Open-graphics mailing list [email protected] http://lists.duskglow.com/mailman/listinfo/open-graphics List service provided by Duskglow Consulting, LLC (www.duskglow.com)
