Isn't this a possibility with most any computer language when one uses binary floating point numbers? Since a real number such as 35.8 can't be EXACTLY represented as a binary fraction, there can always be a bit of floating point fluff added or subtracted, right?
Its analogous to the fact that, for example, 1/3 can't be represented exactly in base 10 -- 0.666 is too low, and 0.667 is too high. If so, then suppose you want to display X digits to the right of the decimal point. If you prepare a number for display by adding a 5 in the X+1 decimal position (e.g. to display 3 decimal digits, you first add 0.0005) then truncating to X digits (e.g. 35.800) you may have your problem solved. (Of course, you want to do this only to values you don't intend to calculate further upon.) If I'm missing the (decimal) point, here, forgive me. Donald Griggs Desk: 803-735-7834 Opinions are not necessarily those of Misys Healthcare Systems nor its board of directors. -----Original Message----- From: Will Leshner [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 13, 2004 1:44 PM To: [EMAIL PROTECTED] Subject: Re: [sqlite] strange rounding problem I've tracked my rounding problem down to this line in vxprintf(): while( realvalue>=10.0 && exp<=350 ){ realvalue *= .1; exp++; } Apparently, on some hardware, sometimes, multiplying a number like '358' by .1 results in a number like '35.799999' or thereabouts. Now what's really strange is that it doesn't happen every time. If I call vxprintf by hand with '358', I get '3.5800000' the first time round that loop. But there is a code path down which I go that causes that value to become strange the first time round the loop. On other hardware the problem never happens, no matter what code path gets to that line. This is with SQLite 2.8.15 built with VC 6. If we go back to our old version of 2.8.6, then we don't have this kind of problem. I don't suppose anybody has ever run into something like this before?
