Okay, so I've got a routine for calculating reciprocal mantissas that uses two tables with 18-bit values, and returns a new 18 bit mantissa. I have a maximum positive error of 1, and a maximum negative error of 0. That is, the result is sometimes 1 too large and sometimes spot on, but it's never too small.
So, theoretically, that is 17 bits precision right? But how do I round this to always get the correct 17 bit mantissa? It seems to be impossible: in ap out aptr aprd 00 00 0 0 0 00 01 0 0 1 01 01 1 0 1 01 10 1 1 1 10 10 1 1 1 10 11 1 1 0 11 11 0 1 0 11 00 0 0 0 where in is the two least significant bits of the correct 18-bit result, ap the two least significant bits of the approximated 18-bit number, out the correct least significant bit of the 17-bit result, aptr the result of truncating the approximation, and aprd the result of rounding the approximation. Seems like both rounding and truncating only give the right answer in three out of four cases. Am I missing something? Lourens _______________________________________________ Open-graphics mailing list [email protected] http://lists.duskglow.com/mailman/listinfo/open-graphics List service provided by Duskglow Consulting, LLC (www.duskglow.com)
