On Jul 31, 2006, at 7:19 AM, Mike Woodworth wrote:
hey all,
i was tracking down a bug in a chunk of declares the other day,
when i found something which took me awhile to understand - i had
written a function to convert rb integers to fixed datatype. it
read like this:
function int2fixed (i as integer) as integer
return i * &hFFFF
end
simple... easy to read... gave the wrong answers?? it appears &h
is returning a double, causing my int to be multiplied in double
space leading to rounding errors. when i switched it to use
bitshiftleft the problems went away.
my question to the list is - now that we have int64, can anyone
conceive of a reason we would want &h to return a double? to the
best of my knowledge &h will always be *trying* to represent an
integer value, correct? is it time to file a feature request? or
is there still some logic to this choice?
It doesn't return a double, at least in 2006r2. If you check with
VarType, it returns either integer(2) or Int64/UInt64(3).
Best,
Jack
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>
Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>
