A wide0 setting of 0x3eb00000 means that any difference of less than 0.000001 between two values is considered insignificant and the two values are considered equal. In your case the difference after OCONV seems to be 0.02 so the values should be considered different
You also imply that the test is performed on internal-format (ie integer) values - if that is the case then it is very unlikely to be a wide0 problem Are you sure that COST.VISIBLE is true ? hth Piers -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Joey Walter Sent: 02 March 2006 00:30 To: [email protected] Subject: [U2] UV: tests for inequality failing. Wide0 parameter? I'm having a problem with comparisons for equality between variables failing. Very simple code: IF COST # COST.WAS THEN IF COST.VISIBLE THEN GOSUB SHOW.COST END And I'm expecting of course for the gosub to be executed when I know for certain the variables COST and COST.WAS are not equal - but it doesn't happen. For instance the test for inequality fails when the variable COST = 1505 and variable COST.WAS = 1507. I wrote a line of code to display the OCONV'd variables and in the case mentioned above it displayed this: recalc.unit.cost; cost was 15.07 cost is 15.05 But, the subroutine to display the new cost did not get executed. Anyway, I'm guessing it's a problem with that bizarre little parameter called Wide0? I currently have it set to 0x3eb00000. Why do I have it set that way - purely because I had problems with the default setting and someone tossed me a clue and told me to try 0x3eb00000 and I never had problems since on my development machine or any other - until now. Anyone that can help me out with this? What's a reasonable/functional setting for Wide0 parameter. I don't need accuracy past six decimal places. Thanks. ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/ ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/
