New topic: Using val() with long integers
<http://forums.realsoftware.com/viewtopic.php?t=37952> Page 1 of 1 [ 2 posts ] Previous topic | Next topic Author Message drewRB Post subject: Using val() with long integersPosted: Fri Mar 04, 2011 7:43 pm Joined: Thu Mar 01, 2007 2:31 pm Posts: 29 I am trying to convert some xml strings that contain large integers and I am getting some odd rounding when using val. For instance: Code:dim x as int64 x = val("6074484029622763686") if inspected x will hold a value of 6074484029622763520. Making x a double will also create the same result. But if i use a variant I get the correct value. Code:dim v as variant dim x as int64 v = "6074484029622763686" x = v.int64value In this case x will correctly equal 6074484029622763686. But if i do the same variant trick with a double I get the wrong value again. Code:dim v as variant dim x as double v = "6074484029622763686" x = v.doublevalue In this case x equals 6074484029622763520. Should I stop using val() and create my own methods that return values converted through variants? Is there a logical reason this behaves as it does? Thanks Top timhare Post subject: Re: Using val() with long integersPosted: Fri Mar 04, 2011 8:04 pm Joined: Fri Jan 06, 2006 3:21 pm Posts: 9386 Location: Portland, OR USA A double only has about 15 digits of precision. Val() returns a double. RB guesses at the appropriate type for any constants in your code and uses the smallest data type that will work. That can throw your calculations off. Code it yourself and be wary of any intermediate values. Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 2 posts ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
