On Jul 31, 2006, at 7:19 AM, Mike Woodworth wrote:
hey all,
i was tracking down a bug in a chunk of declares the other day,
when i found something which took me awhile to understand - i had
written a function to convert rb integers to fixed datatype. it
read like this:
function int2fixed (i as integer) as integer
return i * &hFFFF
end
simple... easy to read... gave the wrong answers?? it appears &h
is returning a double, causing my int to be multiplied in double
space leading to rounding errors.
I doubt that this is the case. Even if the numbers were being treated
as doubles, and I don't know of any way that could happen given the
code you have pasted here, a double can represent any integer up to
2^52 without approximation.
Mars Saxman
REAL Software
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>
Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>
