New topic: Double Trouble
<http://forums.realsoftware.com/viewtopic.php?t=46833> Page 1 of 1 [ 7 posts ] Previous topic | Next topic Author Message Surfer_Joe Post subject: Double TroublePosted: Tue Feb 05, 2013 10:29 pm Joined: Mon Mar 03, 2008 1:29 am Posts: 15 If I take a peek at the following in the debugger myDouble is actually 2275.8499999999999091 not the 2275.8500000000000000 that I expected. Dim myDouble As Double = 2275.85 Maybe I don't truly understand what a double is, If someone could shed some light on this I would appreciate it. Because if I want to evaluate whether or not myDouble = 2275.85 it obviously is not. Thanks for any insight. Joe Top timhare Post subject: Re: Double TroublePosted: Wed Feb 06, 2013 12:09 am Joined: Fri Jan 06, 2006 3:21 pm Posts: 12051 Location: Portland, OR USA While it's not exact, it will at least be consistent. So the comparison will succeed. Top HMARROQUINC Post subject: Re: Double TroublePosted: Wed Feb 06, 2013 1:38 am Joined: Sun Jan 25, 2009 5:11 pm Posts: 413 Location: Guatemala, Central America Yes, the comparison will succeed, for example: dim dbl as double = 2275.85 if dbl = 2275.85 then MsgBox "It is according to RS" + EndOfLine +_ "However, the exact value for dbl is: " + str(dbl, "#.0000000000000000000000000000000000000") end if Will show the msgbox as expected. _________________ Future RS guru. Ride the world! Top kermit Post subject: Re: Double TroublePosted: Wed Feb 06, 2013 2:01 am Joined: Mon May 30, 2011 12:56 am Posts: 601 Its a common 'problem' with doubles: if the two numbers you compare are coming from different sources, or are the result of maths (eg 2275.00 + 0.25) then comparing for equality is unsafe. Many numbers cannot be stored exactly in a double. Especially if the numbers represent money, it is better to convert to some other form before comparing, or to compare 'near enough' conversion (assuming your interest stops at 2 decimal places): if format(x,"0.00") = format(y,"0.00) then .... or if int(x*100) = int(y*100) then Near enough: if abs(x - y) < 0.01 then Top timhare Post subject: Re: Double TroublePosted: Wed Feb 06, 2013 2:07 am Joined: Fri Jan 06, 2006 3:21 pm Posts: 12051 Location: Portland, OR USA Double can represent integer values exactly, up to about 53 bits worth of integer (about a 16 digit number). But it can only exactly represent a small portion of decimal values - those that are a sum of negative powers of two. It represents the fractional portion of the number as the sum of .5, .25, .125, .0625, etc (1/2, 1/4, 1/8, 1/16, ...). Some values can be represented exactly, such as .3125 (.25 + .0625). Most cannot. For those, it keeps adding smaller and smaller fractions until it gets close enough. Top Surfer_Joe Post subject: Re: Double TroublePosted: Wed Feb 06, 2013 1:10 pm Joined: Mon Mar 03, 2008 1:29 am Posts: 15 Thanks for all the replies! I don't like it but at least I understand it. Joe Top DaveS Post subject: Re: Double TroublePosted: Wed Feb 06, 2013 1:43 pm Joined: Sun Aug 05, 2007 10:46 am Posts: 4500 Location: San Diego, CA And so there is no confusion... this is an issue NOT isolated to RealStudio Almost all computer languages that use the IEEE definition of DOUBLE have the same issue, and this include VB.NET _________________ Dave Sisemore MacPro, OSX Lion 10.7.4 RB2012r1 Note : I am not interested in any solutions that involve custom Plug-ins of any kind Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 7 posts ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
