New topic: Rounding decimals again
<http://forums.realsoftware.com/viewtopic.php?t=46074> Page 1 of 1 [ 6 posts ] Previous topic | Next topic Author Message torrid Post subject: Rounding decimals againPosted: Wed Nov 28, 2012 6:46 pm Joined: Sat Jun 04, 2011 8:29 am Posts: 369 I have seen multiple posts about rounding decimals. x = Round(x * 100) / 100 //Round to 2 decimal places I have done this in a piece of code that looks like: anotherReturn = Round(PreReturn * 100)/100 (real variable names changed to protect the innocent) The PreReturn value is set in the app via a grid...I step thru the code and for example see that it is set to 7.5 I then step past this line and anotherReturn = 7.0 Why??? Baffled. Both of these variables are generated using STORM from a sql file and in sql are declared as DECIMAL. _________________ Currently using: Windows XP Pro SP3 Real Studio 2011 Release 4.3 Enterprise and Mac OS 10.7.4 Real Studio 2011 Release 4.3 Enterprise (finally also compiled on Mac) Top ktekinay Post subject: Re: Rounding decimals againPosted: Wed Nov 28, 2012 7:03 pm Joined: Mon Feb 05, 2007 5:21 pm Posts: 274 Location: New York, NY I can't reproduce this so I think we need to see more of your code. I do suggest this change though: anotherReturn = Round(PreReturn * 100.)/100. This will keep all values as doubles instead of converting integers to doubles. _________________ Kem Tekinay MacTechnologies Consulting http://www.mactechnologies.com/ Need to develop, test, and refine regular expressions? Try RegExRX. Top timhare Post subject: Re: Rounding decimals againPosted: Wed Nov 28, 2012 7:09 pm Joined: Fri Jan 06, 2006 3:21 pm Posts: 11849 Location: Portland, OR USA Are both variables defined as double? I threw this into a button dim d, x as double d= 1.2345 x = round(d*100)/100 break and the result is 1.23 as expected. Top torrid Post subject: Re: Rounding decimals againPosted: Wed Nov 28, 2012 8:02 pm Joined: Sat Jun 04, 2011 8:29 am Posts: 369 ktekinay wrote:anotherReturn = Round(PreReturn * 100.)/100. THIS FIXED THE PROBLEM! I would never have thought to do that. Seems obscure. Both variables are defined when STORM converts the .sql code to objects and in there they are DECIMAL which I believe gets converted to DOUBLE. Is using 100. the same as 100.0 ? i.e. RB allows you not to put the zero for some reason... I think a few other languages would barf on that notation. Top ktekinay Post subject: Re: Rounding decimals againPosted: Wed Nov 28, 2012 8:08 pm Joined: Mon Feb 05, 2007 5:21 pm Posts: 274 Location: New York, NY Yes, that's the same. Either will define it as a double constant. _________________ Kem Tekinay MacTechnologies Consulting http://www.mactechnologies.com/ Need to develop, test, and refine regular expressions? Try RegExRX. Top timhare Post subject: Re: Rounding decimals againPosted: Wed Nov 28, 2012 8:15 pm Joined: Fri Jan 06, 2006 3:21 pm Posts: 11849 Location: Portland, OR USA torrid wrote:ktekinay wrote:anotherReturn = Round(PreReturn * 100.)/100. THIS FIXED THE PROBLEM! I would never have thought to do that. Seems obscure. Both variables are defined when STORM converts the .sql code to objects and in there they are DECIMAL which I believe gets converted to DOUBLE. It would not appear to be the case. If adding the decimal point fixed it, then PreReturn is some kind of integer value. Maybe Currency? You should find out. Quote:Is using 100. the same as 100.0 ? i.e. RB allows you not to put the zero for some reason... I think a few other languages would barf on that notation. Yes, they are the same. Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 6 posts ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
