Re: [sqlite] error in round-function?
--- [EMAIL PROTECTED] wrote: > Going into this statement on both Linux and Windows, > the exact same values are in realvalue and in rounder: > >realvalue: 0xf000 0.94995559107901499373838 >rounder:0x3faa 0.0500030 > > Both machines execute the exact same 4 ix86 opcodes: > >fldl >fldt >faddp %st,%st(1) >fstpt > > But they end up with different answers. Linux gets > what I believe is the correct answer: > >realvalue: 0xfd00 0.5836663657655662973 > > Windows gets the wrong answer: > >realvalue: 0x8000 1.000 > > Can somebody please suggest to me what the difference is? > Does windows somehow initialize the floating point hardware > differently so that it does some kind of rounding on its own? > Why should the same machine-language instructions generate a > different answer on windows than it does on linux? http://support.microsoft.com/kb/102555 Microsoft Visual C++ runtime library provides default floating-point exception handling and includes functions such as _controlfp for determining and adjusting the floating-point hardware's rounding, precision control, and exception handling behavior. More info on fp:precise and _controlfp: http://msdn2.microsoft.com/en-us/library/aa289157(vs.71).aspx double a, b, cLower, cUpper; . . . _controlfp( _RC_DOWN, _MCW_RC );// round to - cLower = a*b; _controlfp( _RC_UP, _MCW_RC );// round to + cUpper = a*b; _controlfp( _RC_NEAR, _MCW_RC );// restore rounding mode Now that's room service! Choose from over 150,000 hotels in 45,000 destinations on Yahoo! Travel to find your fit. http://farechase.yahoo.com/promo-generic-14795097 - To unsubscribe, send email to [EMAIL PROTECTED] -
RE: [sqlite] error in round-function?
Just checked my code and luckily I don't round in SQLite. I suppose an easy work-around for now would be to do something like: Select round(field + 0.001, 1) as it will be unlikely you are dealing with 0.949 RBS -Original Message- From: Olaf Schmidt [mailto:[EMAIL PROTECTED] Sent: 10 June 2007 21:34 To: sqlite-users@sqlite.org Subject: Re: [sqlite] error in round-function? Thanks to RBS and Alberto for testing... So it seems the problem is somehow Windows-related (maybe the VC-Compiler). Think I'll override the Round-Function in my wrapper. Regards, Olaf -- View this message in context: http://www.nabble.com/error-in-round-function--tf3897765.html#a11052069 Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] - - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
--- Olaf Schmidt <[EMAIL PROTECTED]> wrote: > On what OS have you tested? > If on windows, was it a GCC-compile or a MS-VC-compile? Running the GCC cross-compiled sqlite3.exe from http://www.sqlite.org/sqlite-3_3_17.zip on Windows (well, wine on Linux): SQLite version 3.3.17 Enter ".help" for instructions sqlite> select round(0.95, 1); 0.9 What result does the sqlite3.exe from the above link give you on your machine? Shape Yahoo! in your own image. Join our Network Research Panel today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
Olaf Schmidt <[EMAIL PROTECTED]> wrote: > > select round(0.95, 1) > gives 0 - not 0.9 (in case rounding down) and > also not 1.0 (in case rounding up) > I can reproduce the problem by running sqlite3.exe on windows. But I cannot explain it. The problem occurs in printf.c on line 454: if( xtype==etFLOAT ) realvalue += rounder; Going into this statement on both Linux and Windows, the exact same values are in realvalue and in rounder: realvalue: 0xf000 0.94995559107901499373838 rounder:0x3faa 0.0500030 Both machines execute the exact same 4 ix86 opcodes: fldl fldt faddp %st,%st(1) fstpt But they end up with different answers. Linux gets what I believe is the correct answer: realvalue: 0xfd00 0.5836663657655662973 Windows gets the wrong answer: realvalue: 0x8000 1.000 Can somebody please suggest to me what the difference is? Does windows somehow initialize the floating point hardware differently so that it does some kind of rounding on its own? Why should the same machine-language instructions generate a different answer on windows than it does on linux? -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
Thanks to RBS and Alberto for testing... So it seems the problem is somehow Windows-related (maybe the VC-Compiler). Think I'll override the Round-Function in my wrapper. Regards, Olaf -- View this message in context: http://www.nabble.com/error-in-round-function--tf3897765.html#a11052069 Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
On 6/10/07, Olaf Schmidt <[EMAIL PROTECTED]> wrote: > > select round(0.95, 1) > 0.9 > > select round(9.95, 1) > 9.9 > > select round(0.995, 2) > 0.99 > > select round(9.995, 2) > 9.99 > (3.3.17 here) As it should be, hmm. On what OS have you tested? If on windows, was it a GCC-compile or a MS-VC-compile? MacOS X, gcc 4.0.1 Cheers -- Alberto Simões - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
> > select round(0.95, 1) > 0.9 > > select round(9.95, 1) > 9.9 > > select round(0.995, 2) > 0.99 > > select round(9.995, 2) > 9.99 > (3.3.17 here) As it should be, hmm. On what OS have you tested? If on windows, was it a GCC-compile or a MS-VC-compile? Olaf -- View this message in context: http://www.nabble.com/error-in-round-function--tf3897765.html#a11051093 Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
On 6/10/07, Olaf Schmidt <[EMAIL PROTECTED]> wrote: Hi drh, > http://www.sqlite.org/faq.html#q18 Not so easy I think (I'm well aware of rounding-problems regarding the IEEE-Float-Formats, etc.). Bankers rounding aside at the moment. select round(0.95, 1) 0.9 select round(9.95, 1) 9.9 select round(0.995, 2) 0.99 select round(9.995, 2) 9.99 (3.3.17 here) -- Alberto Simões - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
Hi drh, > http://www.sqlite.org/faq.html#q18 Not so easy I think (I'm well aware of rounding-problems regarding the IEEE-Float-Formats, etc.). Bankers rounding aside at the moment. select round(0.95, 1) gives 0 - not 0.9 (in case rounding down) and also not 1.0 (in case rounding up) Funny thing, just tested with the example in the FAQ. select round(9.95, 1) gives 0 too, not 9.9 and also not 10.0. The wrong Zero-Output occurs also with the following: select round(0.995, 2) select round(9.995, 2) Could someone test this with some newer SQLite-binary. Maybe the error has something to do with my own compiler- settings or maybe occurs on Windows only. I'm using a "StdCall-Dll" here (3.3.17, compiled with VC8) under XP. Olaf -- View this message in context: http://www.nabble.com/error-in-round-function--tf3897765.html#a11050747 Sent from the SQLite mailing list archive at Nabble.com. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
Olaf Schmidt <[EMAIL PROTECTED]> wrote: > The integrated round-function works well so far, > especially because it does "bankers rounding". > Tough it seems to have a little bug, wich is probably simple to fix. > > Try... > select round(0.94, 1) -> gives 0.9, wich is correct > select round(0.96, 1) -> gives 1, wich is correct too > > but... > select round(0.95, 1) -> gives 0, wich is not correct of course > Regarding bankers rounding, wich rounds up in case of a > '5' at the appropriate position, the correct result should be 1. > > (tested with 3.3.17) I do not know what are you trying to round. In any case, I just can say that if you are storing a column as a float for money, turn it into a string. This is what I've learned when writing Perl code to deal with money :) Cheers Alberto -- Alberto Simões - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] error in round-function?
Olaf Schmidt <[EMAIL PROTECTED]> wrote: > The integrated round-function works well so far, > especially because it does "bankers rounding". > Tough it seems to have a little bug, wich is probably simple to fix. > > Try... > select round(0.94, 1) -> gives 0.9, wich is correct > select round(0.96, 1) -> gives 1, wich is correct too > > but... > select round(0.95, 1) -> gives 0, wich is not correct of course > Regarding bankers rounding, wich rounds up in case of a > '5' at the appropriate position, the correct result should be 1. > > (tested with 3.3.17) > http://www.sqlite.org/faq.html#q18 -- D. Richard Hipp <[EMAIL PROTECTED]> - To unsubscribe, send email to [EMAIL PROTECTED] -
