New topic: losing precision in maths somewhere?? (or my mind)
<http://forums.realsoftware.com/viewtopic.php?t=33858> Page 1 of 1 [ 1 post ] Previous topic | Next topic Author Message gokartracer101 Post subject: losing precision in maths somewhere?? (or my mind)Posted: Wed May 19, 2010 10:06 am Joined: Tue Sep 22, 2009 2:18 pm Posts: 9 Hi I'm receiving some unexpected results, and I can't work out where the calculation is going wrong. For example, I'm passing the value "46.106" and I want the final output to read in the format "0:46.106" and so forth.... however, the value being returned is "0:46.105"; somewhere I'm losing 0.001 of the value!!!!!!!! I think I've been looking at this too long, and simply can't see my own face in the mirror anymore. I'm sure it'll be simple in the end, but I just can't think of how to get around this for now? I have the following code: Function formatTime (rawTime as double) as String dim minutes, seconds as integer dim fractions as double dim strMin, strFrac as string minutes = floor(rawTime/60) seconds = floor(rawTime mod 60) strMin = format(seconds, "00") fractions = (rawTime * 1000) mod 1000 strFrac = format(fractions, "000") dim result as string result = str(minutes) + ":" + strMin + "." + strFrac return result any suggestions please? and many thanks in advance!! Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 1 post ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
