New topic: Weird Integer/FP Overflow
<http://forums.realsoftware.com/viewtopic.php?t=37034> Page 1 of 1 [ 1 post ] Previous topic | Next topic Author Message charonn0 Post subject: Weird Integer/FP OverflowPosted: Tue Jan 11, 2011 2:48 pm Joined: Mon Apr 02, 2007 2:08 am Posts: 282 Location: San Francisco, CA, USA Code:Function foo() Dim mbsize As UInt64 = 80000000 Dim pLen As Integer = 5000 Dim count As UInt64 = 0 Dim percent As UInt64 = 0 For x As Integer = 0 To pLen percent = (count * 100) \ (mbsize / 3) If percent < 0 Then Break End If For y As Integer = 0 To pLen Count = Count + 1 Next Next End Function If you run the above code, it will break into the debugger on the 4,295th iteration of the x loop and percent will be 18,446,744,073,709,551,536 which is just 79 lower than the largest number that can be stored in a UInt64 (and rather a lot larger than zero) However, this code works as expected: Code:Function foo() Dim mbsize As UInt64 = 80000000 Dim pLen As Integer = 5000 Dim count As UInt64 = 0 Dim percent As UInt64 = 0 For x As Integer = 0 To pLen percent = (count * 100) / (mbsize / 3) If percent < 0 Then Break End If For y As Integer = 0 To pLen Count = Count + 1 Next Next End Function Did you see the difference? "percent = (count * 100) / (mbsize / 3)" The only change is to what type of division is used, integer or floating point. So, is this a bug in RB's handling of integer division or is it another Intel FDIV bug? _________________ Boredom Software Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 1 post ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
