look up the ECL statement FLOAT.PRECISION Folks are often happiest with FLOAT.PRECISION 4
Wally Terhune U2 Support Architect IBM Information Management Software Tel: (303) 773-7969 T/L 656-7969 Mobile: (303) 807-6222 Email: [email protected] http://www.ibm.com/software/data/u2/support From: "Martin Hutchinson" <[email protected]> To: "u2 user group" <[email protected]> Date: 05/01/2009 05:21 AM Subject: [U2] Unidata calculations incorrect Sent by: [email protected] Hi All, We have Unidata 7.2 running on Windows 2008 server making incorrect calculations when numbers are close to zero. I understand there is a 'WIDEZERO' parameter on Univers but I don't think there is an equivalent on Unidata. Below we have an example program and output. 001: BAL = 856.19 002: VAL1 = 287.31 003: VAL2 = 568.88 004: BAL = BAL - VAL1 005: BAL = BAL - VAL2 006: * 007: CRT "BAL [":BAL:"]" 008: * 009: IF BAL > 0 THEN 010: CRT "BAL > 0" 011: END ELSE 012: CRT "BAL <= 0" 013: END 014: * 015: IF INT(BAL) = BAL THEN 016: CRT "Matches integer" 017: END ELSE 018: CRT "Does not match integer" 019: END 020: * 021: IF ICONV(BAL,"MCN") > 0 THEN 022: CRT "MCN BAL > 0" 023: END ELSE 024: CRT "MCN BAL <= 0" 025: END When run this gives: BAL [0] BAL > 0 Does not match integer MCN BAL <= 0 Has anyone come across this one or have any ideas? regards Martin Hutchinson ________________________ Web Control (UK) Ltd Atlantic House, Michigan Drive, Tongwell, Milton Keynes, MK15 8HQ T: +44(0)1908 216700 x4 F: +44(0)1908 216300 http://www.webcontrolltd.co.uk Web Control (UK) Ltd is a company registered in England and Wales with company number 4118558. VAT number: 770 3363 38 ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/ [demime 1.01d removed an attachment of type image/gif which had a name of graycol.gif] [demime 1.01d removed an attachment of type image/gif which had a name of ecblank.gif] ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/
