Do you have a reference for this? I found 3:
Wikipedia says 16 http://en.wikipedia.org/wiki/IEEE_754-2008 BYU says 15 http://www.math.byu.edu/~schow/work/IEEEFloatingPoint.htm Oracle says 15-17 http://docs.oracle.com/cd/E19957-01/806-3568/ncg_math.html But I've never heard of "expected precision" and google doesn't come up with term either -- it either has precision or it doesn't as far as I've ever heard. And i've never heard of 14 digits before (which wouidn't surprise me though). Michael D. Black Senior Scientist Advanced Analytics Directorate Advanced GEOINT Solutions Operating Unit Northrop Grumman Information Systems ________________________________ From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Keith Medcalf [kmedc...@dessus.com] Sent: Saturday, June 16, 2012 8:11 PM To: General Discussion of SQLite Database Subject: EXT :Re: [sqlite] VERY weird rounding error > Ideally, you write your software so that differences in the 15th decimal > place don't matter to you. If you want two platforms to give identical > results you use integer arithmetic. It really depends on what kind of > application you're writing (scientific, financial) or what your numbers > actually represent. This is technically incorrect. Floating Point accuracy is in Decimal Digits, not Decimal Places (unless, of course, the decimal formatted output is in "Engineering Notation" in which case Decimal Digits == Decimal Places -- Engineering Notation means that the number to the left of the decimal point is 0). The "theoretical" accuracy (and the "reasonably expected" accuracy) is: Half (2 Bytes) ~ 3.3 2 Single (4 Bytes) ~ 7.2 6 Double (8 bytes) ~15.9 14 Double extended (10 Byte) ~19.2 18 Quad ( 16 Bytes) ~34.0 32 The value 9990.1 has 5 significant digits. The output value 99990.1000000001 has 15 significant digits, of which the expected 14 are accurate. Etienne, thanks for the thanks. --- () ascii ribbon campaign against html e-mail /\ www.asciiribbon.org<http://www.asciiribbon.org/> > -----Original Message----- > From: sqlite-users-boun...@sqlite.org [<thismessage:/>mailto:sqlite-users- > boun...@sqlite.org] On Behalf Of Simon Slavin > Sent: Saturday, 16 June, 2012 17:33 > To: General Discussion of SQLite Database > Subject: Re: [sqlite] VERY weird rounding error > > > On 17 Jun 2012, at 12:06am, Etienne <ejlist-sql...@yahoo.fr> wrote: > > > Is there a trick (in the sys. libraries mentioned above, or through the > win32 C API) for changing the way the FPU handles (long) doubles? > > You're at tricky low level detail now. Rounding and truncation modes used by > chips which implement IEEE754 are often carefully set by compiler writers to > get the results they want. If you mess with the settings they've chosen, you > can get unexpected results in other apps which were previously working > correctly. > > > It's discovering and caring about this sort of thing that separates out the > professional programmer from the someone who shouldn't be paid for > programming. > > I do appreciate your thanks, by the way. > > Simon. > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
