It's all about binary arithmetic. Spreadsheets have traditionally had this same 
issue. There are workarounds for it, but there's a substantial performance 
penalty and in most cases it's simply not enough to worry about. The number is 
always only off by one bit at the limit of your precision. 

> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:dqsd-users-
> [EMAIL PROTECTED] On Behalf Of Kim Gräsman
> Sent: May-17-07 10:09 PM
> To: DQSD users mailing list
> Subject: Re: [DQSD-Users] Odd Math Error
> 
> Hi Richard,
> 
> > Uhhh that doesnt sound right to me. I subtracted 3 different values
> > from another value and got the same result twice.
> 
> Oops, I didn't see your results were identical for the first two cases.
> I tried it here;
> 
> 2156737.51 - 16123.81 => 2140613.6999999997
> 2156737.51 - 16123.83 => 2140613.6799999997 <-- Note: .6_7_999...
> 2156737.51 - 16123.82 => 2140613.69
> 
> Could you try it again to make sure you transcribed the values correctly?
> 
> > Also two of the results are flat out wrong.
> 
> Right, but they're pretty close to correct, which is what you get with
> floating-point, if I understand things correctly.
> 
> > I could understand if I had used some kind of hokey notation but its
> > a straight up math problem X-Y. When performed in a calculator
> > application I dont get that result?
> 
> The calculator (at least the one in Windows) probably doesn't use
> floating-point to store its numbers, but again, I'm a bit out of my
> depth here.
> 
> Cheers,
> - Kim
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> To unsubscribe visit:
> https://lists.sourceforge.net/lists/listinfo/dqsd-users
> DQSD-Users@lists.sourceforge.net
> http://sourceforge.net/mailarchive/forum.php?forum_id=8601

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
To unsubscribe visit:
https://lists.sourceforge.net/lists/listinfo/dqsd-users
DQSD-Users@lists.sourceforge.net
http://sourceforge.net/mailarchive/forum.php?forum_id=8601

Reply via email to