New topic: Decimal Values
<http://forums.realsoftware.com/viewtopic.php?t=45721> Page 1 of 1 [ 9 posts ] Previous topic | Next topic Author Message tseyfarth Post subject: Decimal ValuesPosted: Thu Oct 25, 2012 3:34 pm Joined: Sat Dec 04, 2010 9:14 pm Posts: 710 Hello All. I have a routine that finds a battery voltage. The voltage is a value that is hard coded. However, when I look at the debugger, the variable, dimensioned as a single, does not read the value it should. Instead it reads slightly less. Instead of 2.4 it reads 2.399999999999999 ...... How to correct this? Thanks Tim Top timhare Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 3:45 pm Joined: Fri Jan 06, 2006 3:21 pm Posts: 11751 Location: Portland, OR USA Nothing to correct. The debugger is displaying the full value, which for a fractional value will rarely be exact. If you format it to 1 or 2 decimal places, you will get 2.4. Top npalardy Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 3:53 pm Real Software Engineer Joined: Sat Dec 24, 2005 8:18 pm Posts: 7508 Location: Canada, Alberta, Near Red Deer tseyfarth wrote:Hello All. I have a routine that finds a battery voltage. The voltage is a value that is hard coded. However, when I look at the debugger, the variable, dimensioned as a single, does not read the value it should. Instead it reads slightly less. Instead of 2.4 it reads 2.399999999999999 ...... How to correct this? Thanks Tim http://en.wikipedia.org/wiki/Floating_point _________________ My web site Great White Software RBLibrary.com REALbasic learning Top Jason_Adams Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 7:09 pm Joined: Fri Nov 10, 2006 4:10 pm Posts: 1690 Location: Michigan, USA On top of what they're saying, if you actually convert the value using Str() or Format(), you'll get the 2.4. What you're seeing is the debugger taking the StringValue of a Variant containing a Single â not the recommended method of converting a value to string. Hope this helps! _________________ Windows 7 Ultimate x64 Windows XP Pro SP3 Ubuntu 11.04 via Virtual Box RS Enterprise 2012r1.1 Programming Tutorials & Free Projects: http://www.JasonTheAdams.com "Christianity has not been tried and found wanting; it has been found difficult and not tried." - G.K. Chesterton Top tseyfarth Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 7:32 pm Joined: Sat Dec 04, 2010 9:14 pm Posts: 710 Thanks all, I was not trying to convert a value to a string at all. I have a byte received from a device. Each bit from bit7 to bit0 represents a battery voltage. So if bit7 is High, then the voltage is 3.0. It descends from there: For i As Integer =8 DownTo 1 If i = 1 then 'Bit 0 2.3 If BitOn(ByteData(2), i) = True Then IW.BattVoltage = "2.30" Exit For End If ElseIf i = 2 Then 'Bit 1 2.4 If BitOn(ByteData(2), i) = True Then IW.BattVoltage = "2.40" Exit For End If ElseIf i = 3 Then 'Bit 2 2.50 If BitOn(ByteData(2), i) = True Then IW.BattVoltage ="2.50" Exit For End If ElseIf i = 4 Then 'Bit 3 2.6 If BitOn(ByteData(2), i) = True Then IW.BattVoltage = "2.60" Exit For End If ElseIf i = 5 Then 'Bit 4 2.7 If BitOn(ByteData(2), i) = True Then IW.BattVoltage = "2.70" Exit For End If ElseIf i = 6 Then 'Bit 5 2.8 If BitOn(ByteData(2), i) = True Then IW.BattVoltage = "2.80" Exit For End If ElseIf i = 7 Then 'Bit 6 2.9 If BitOn(ByteData(2), i) = True Then IW.BattVoltage = "2.90" Exit For End If ElseIf i = 8 Then'Bit 7 3 If BitOn(ByteData(2), i) = True Then IW.BattVoltage = "3.0" Exit For End If End If Next i Note, the above has been converted TO work with strings. Previously the variable, IW.BattVoltage was a Single. Since the value was hard coded, I was surprised to see that the value was not exactly AS coded. Tim Top DaveS Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 8:31 pm Joined: Sun Aug 05, 2007 10:46 am Posts: 4197 Location: San Diego, CA here is an idea. [NOT TESTED] Dim v As Integer If x>0 Then v=22 Do v=v+1 If (x And 1)=1 Then Exit Do x=Bitwise.Shiftleft(x,1) Loop Else v=0 End If it assumes that no more than ONE bit will be set, and uses an INTEGER to avoid rounding (returns 0 or 23 to 30) _________________ Dave Sisemore MacPro, OSX Lion 10.7.4 RB2012r1 Note : I am not interested in any solutions that involve custom Plug-ins of any kind Top tseyfarth Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 9:30 pm Joined: Sat Dec 04, 2010 9:14 pm Posts: 710 Dave, That will not work since the batt value is based on the highest bit setting. Note the batt value declines as more bits are set to 0. I was only wanting to store the value as an number, instead of a string. But tonite, not something to quibble about too much. Thank you for responding however! Tim Top ktekinay Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 11:00 pm Joined: Mon Feb 05, 2007 5:21 pm Posts: 214 Location: New York, NY If I may... dim i as integer = 8 do if BitOn( ByteData( 2 ), i ) then exit i = i - 1 loop until i = 0 dim voltage as single if i = 0 then // No bits set -- what then? else voltage = ( 22 + i ) / 10. end if return format( voltage, "0.00" ) If the conversion from single to string is giving you trouble, try this: static voltageChart() as string if voltageChart.Ubound = -1 then // Only want to set this once voltageChart = Array( "", "2.3", "2.4", "2.5", "2.6", "2.7", "2.8", "2.9", "3.0" ) end if dim voltage as string dim i as integer = 8 do if BitOn( ByteData( 2 ), i ) then voltage = voltageChart( i ) exit end if i = i - 1 loop until i = 0 if i = 0 then // No bits set -- what then? end if return voltage This is off the top of my head so check for bugs. _________________ Kem Tekinay MacTechnologies Consulting http://www.mactechnologies.com/ Need to develop, test, and refine regular expressions? Try RegExRX. Top tseyfarth Post subject: Re: Decimal ValuesPosted: Thu Oct 25, 2012 11:18 pm Joined: Sat Dec 04, 2010 9:14 pm Posts: 710 If no bits set, voltage is < 2.3 and probably would not be transmitted anyway. I was not trying to convert to a string. I wanted a real number value. I did not expect 2.3 to be 2.29999999999999999999 - when using a variable type of Single. That was the whole problem. So, to get around the problem of a value of 2.3 being presented as 2.299999999 I just switched to using a String. Interesting however, is your use of a "Chart". I will look at that for other applications, and maybe this one too. The code works as is. It was intended to not be cumulative, in terms of how many bits are on. Instead, it was developed as a scale where each bit on, represented a floor voltage value. This allowed me to obtain data from a wireless transceiver board, which I used as part of a new product. Since I have total control of the system (I developed End To End the firmware, and hardware that RS talks to) I did not have to worry about anything other than my specific purposes. Conveniently, it also matches the output data from the radio chip. If you take another look at it: b0 = 2.3V b1 = 2.4V b2 = 2.5V b3 = 2.6V b4 = 2.7V b5 = 2.8V b6 = 2.9V b7 = 3.0V If all are 0, it either will not transmit, or will transmit that last time, since the threashold IS 2.3V. Conversely, if b4 is a 1, then I do not have to look any further. The voltage is 2.7. Thanks again for all of the ideas here. But again, my issue was not going to a string, but rather keeping the value as what it should be, and not .00000000000000000001 less..... Tim Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 9 posts ]
-- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
