New topic: Math problem with different variable types
<http://forums.realsoftware.com/viewtopic.php?t=31409> Page 1 of 1 [ 2 posts ] Previous topic | Next topic Author Message dgdavidge Post subject: Math problem with different variable typesPosted: Thu Dec 03, 2009 7:50 pm Joined: Fri Jun 02, 2006 1:43 pm Posts: 57 Location: Santa Ynez, CA I have a formula that uses variables from different sources with the variables being different types it works fine if one of the integer variables = 6400 (result is about 7,000,000), but if that variable is 48,000, the result is a twenty digit number. The formula is Code:UInt64 = UInt32 * Uint32 * Integer * Single / 512 / Integer The final Integer is the one that changes from 6400 to 48000 and crashes everything. I'm tempted to change everything to Int64s and brute force it, but that sounds like it overkill. The single only has two digits to the right of the decimal so it could be converted to an integer easily. Can someone enlighten me on how RB does math and what is the best approach? I am talking to a controller over a serial port and receiving UInt32 values and having to send a UInt64. Top npalardy Post subject: Re: Math problem with different variable typesPosted: Thu Dec 03, 2009 10:21 pm Joined: Sat Dec 24, 2005 8:18 pm Posts: 5455 Location: Canada, Alberta, Near Red Deer When you say "crashes" you mean RB crashes ? or ? It's possible that UInt32 * Uint32 * Integer * Single / 512 / Integer is generating a SINGLE as the intermediate type RB does NOT look at the left hand side of the assignment to see what type to generate or what intermediate types to use It looks at the types that are part of the statement so it's probably generating a SINGLE as a value and that's possibly overflowing _________________ My web site Great White Software RBLibrary.com REALbasic learning Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 2 posts ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
