Has anyone figured out exactly how much precision we actually NEED?
On Tue, 15 Feb 2005 16:53:53 +0100, Lourens Veen <[EMAIL PROTECTED]> wrote: > Okay, so I've got a routine for calculating reciprocal mantissas that uses two > tables with 18-bit values, and returns a new 18 bit mantissa. I have a > maximum positive error of 1, and a maximum negative error of 0. That is, the > result is sometimes 1 too large and sometimes spot on, but it's never too > small. > > So, theoretically, that is 17 bits precision right? But how do I round this to > always get the correct 17 bit mantissa? It seems to be impossible: > > in ap out aptr aprd > 00 00 0 0 0 > 00 01 0 0 1 > 01 01 1 0 1 > 01 10 1 1 1 > 10 10 1 1 1 > 10 11 1 1 0 > 11 11 0 1 0 > 11 00 0 0 0 > > where in is the two least significant bits of the correct 18-bit result, ap > the two least significant bits of the approximated 18-bit number, out the > correct least significant bit of the 17-bit result, aptr the result of > truncating the approximation, and aprd the result of rounding the > approximation. > > Seems like both rounding and truncating only give the right answer in three > out of four cases. > > Am I missing something? > > Lourens > _______________________________________________ > Open-graphics mailing list > [email protected] > http://lists.duskglow.com/mailman/listinfo/open-graphics > List service provided by Duskglow Consulting, LLC (www.duskglow.com) > _______________________________________________ Open-graphics mailing list [email protected] http://lists.duskglow.com/mailman/listinfo/open-graphics List service provided by Duskglow Consulting, LLC (www.duskglow.com)
