Inside a Flex Builder 3 Actionscript project, when debugging..
Code:
var n1:int=Math.abs(int.MAX_VALUE - 1290);
var n2:int=Math.abs(int.MAX_VALUE - 1284);
var n3:Number=Math.abs(int.MAX_VALUE - 1290);
var n4:Number=Math.abs(int.MAX_VALUE - 1284);
if (Math.abs(int.MAX_VALUE - 1290) == Math.abs(int.MAX_VALUE - 1284)) {
var nh:int=2; // with a breakpoints here to see if the
nh++; // condition is true. It never arrived here.
}
In the variables pane:
n1 2147482357 [0x7ffffaf5]
n2 2147482363 [0x7ffffafb]
n3 2147482357 [0x7ffffaf5]
n4 2147482363 [0x7ffffafb]
Everything works as expected.
John
----- Original Message -----
From: Daniel Freiman
To: [email protected]
Sent: Friday, June 06, 2008 10:10 PM
Subject: Re: [flexcoders] Math.abs() Limitation?
I figured it out. The trace made it obvious (as opposed the debugger which
is what I was previously using). I was using Number.MAX_VALUE, not
int.MAX_VALUE. Number.MAX_VALUE doesn't have enough precision to handle what I
was trying to do.
Thanks,
- Daniel Freiman
On Fri, Jun 6, 2008 at 2:29 PM, Gordon Smith <[EMAIL PROTECTED]> wrote:
What do you get when you trace out the following values?
int.MAX_VALUE - 1290
Math.abs(int.MAX_VALUE - 1290)
int.MAX_VALUE - 1284
Math.abs(int.MAX_VALUE - 1284)
Gordon Smith
Adobe Flex SDK Team
----------------------------------------------------------------------------
From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of
Alex Harui
Sent: Friday, June 06, 2008 11:18 AM
To: [email protected]
Subject: RE: [flexcoders] Math.abs() Limitation?
What if you use temporary variables?
----------------------------------------------------------------------------
From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of
Daniel Freiman
Sent: Friday, June 06, 2008 9:09 AM
To: flexcoders
Subject: [flexcoders] Math.abs() Limitation?
According to my code the following statement returns true.
Math.abs(int.MAX_VALUE - 1290) == Math.abs(int.MAX_VALUE - 1284)
I tried converting everything to type Number that didn't help. Getting
ride of the abs() makes the calculation work correctly but then I don't have
the absolute value.
Thoughts?
- Daniel Freiman
