Re: [U2] The math just doesn't work.

2011-10-11 Thread Martin Phillips
@listserver.u2ug.org Sent: Monday, October 10, 2011 6:35 PM Subject: Re: [U2] The math just doesn't work. I thought that large integer addition is why the functions SADD, SCMP and SDIV were invented? ___ U2-Users mailing list U2-Users@listserver.u2ug.org

Re: [U2] The math just doesn't work.

2011-10-11 Thread Brett Callacher
- Original Message - *From:* bre...@gpmdev.co.uk *To:* u2-users@listserver.u2ug.org u2-users@listserver.u2ug.org *Date:* 10/10/2011 1:16 AM *Subject:* Re: [U2] The math just doesn't work. I also thought: great explanation and nodded sagely when I read this. However

Re: [U2] The math just doesn't work.

2011-10-10 Thread Brett Callacher
I also thought: great explanation and nodded sagely when I read this. However, consider this code: NUL = '' CM = ',' A = -409071.8775: CM: 475000: CM: -652413: CM: 652413: CM: -475000: CM: 409071.8775 * * TEST.VMC = 1 TEST.TOTAL = 0 * LOOP UNTIL

Re: [U2] The math just doesn't work.

2011-10-10 Thread Martin Phillips
Coldstream Lane, Hardingstone, Northampton NN4 6DB, England +44 (0)1604-709200 - Original Message - From: Brett Callacher bre...@gpmdev.co.uk To: u2-users@listserver.u2ug.org Sent: Monday, October 10, 2011 9:16 AM Subject: Re: [U2] The math just doesn't work. I also thought: great

Re: [U2] The math just doesn't work.

2011-10-10 Thread Brett Callacher
- From: Brett Callacher bre...@gpmdev.co.uk To: u2-users@listserver.u2ug.org Sent: Monday, October 10, 2011 9:16 AM Subject: Re: [U2] The math just doesn't work. I also thought: great explanation and nodded sagely when I read this. However, consider this code: NUL

Re: [U2] The math just doesn't work.

2011-10-10 Thread Martin Phillips
Hi Brett, The realisation for me here is that PRECISION has nothing to do with the precision of the calculation, it only effects display. Correct. All arithetic operations work at the precision of the underlying hardware. PRECISION applies when converting a floating point number to a

Re: [U2] The math just doesn't work.

2011-10-10 Thread Israel, John R.
List Subject: Re: [U2] The math just doesn't work. Hi Brett, The realisation for me here is that PRECISION has nothing to do with the precision of the calculation, it only effects display. Correct. All arithetic operations work at the precision of the underlying hardware. PRECISION applies when

Re: [U2] The math just doesn't work.

2011-10-10 Thread Allen Egerton
On 10/10/2011 8:12 AM, Israel, John R. wrote: I am coming late to this party, but wanted to throw out a bug that was just reported to Rocket. We found that PERCISION was causing a calculation to round in some cases and truncate in others. I am not sure where this stands because someone

Re: [U2] The math just doesn't work.

2011-10-10 Thread Israel, John R.
Egerton Sent: Monday, October 10, 2011 8:31 AM To: U2 Users List Subject: Re: [U2] The math just doesn't work. On 10/10/2011 8:12 AM, Israel, John R. wrote: I am coming late to this party, but wanted to throw out a bug that was just reported to Rocket. We found that PERCISION was causing

Re: [U2] The math just doesn't work.

2011-10-10 Thread Wols Lists
On 10/10/11 13:31, Allen Egerton wrote: On 10/10/2011 8:12 AM, Israel, John R. wrote: I am coming late to this party, but wanted to throw out a bug that was just reported to Rocket. We found that PERCISION was causing a calculation to round in some cases and truncate in others. I am not

Re: [U2] The math just doesn't work.

2011-10-10 Thread Allen Egerton
On 10/10/2011 12:31 PM, Wols Lists wrote: On 10/10/11 13:31, Allen Egerton wrote: On 10/10/2011 8:12 AM, Israel, John R. wrote: I am coming late to this party, but wanted to throw out a bug that was just reported to Rocket. We found that PERCISION was causing a calculation to round in some

Re: [U2] The math just doesn't work.

2011-10-10 Thread Bill Haskett
- Original Message - *From:* bre...@gpmdev.co.uk *To:* u2-users@listserver.u2ug.org u2-users@listserver.u2ug.org *Date:* 10/10/2011 1:16 AM *Subject:* Re: [U2] The math just doesn't work. I also thought: great explanation and nodded sagely when I read this. However

Re: [U2] The math just doesn't work.

2011-09-30 Thread Martin Phillips
Hi Mecki, I actually wonder if this feature is flavour specific? Pick didn't use floating point arithmetic and IIRC the default precision used to be 4. PRECISION affects the number of decimal places in the result when converting a floating point value to a character string. All internal

Re: [U2] The math just doesn't work.

2011-09-29 Thread Martin Braid
ICONV everything MD2 and check that for equality -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Hammerle Sent: 29 September 2011 13:30 To: u2-users@listserver.u2ug.org Subject: [U2] The math just doesn't

Re: [U2] The math just doesn't work.

2011-09-29 Thread Martin Phillips
Hi George, As a general rule in programming, comparison of floating point values for equality should be avoided. This is because, just as we cannot write the number one third accurately in decimal notation, so the IEEE floating point format used by computer systems cannot store numbers

Re: [U2] The math just doesn't work.

2011-09-29 Thread Brian Leach
= 4203.09, C= 4203.09 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin Phillips Sent: 29 September 2011 13:46 To: U2 Users List Subject: Re: [U2] The math just doesn't work. Hi George, As a general rule

Re: [U2] The math just doesn't work.

2011-09-29 Thread Oaks, Harold
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Brian Leach Sent: Thursday, September 29, 2011 6:21 AM To: 'U2 Users List' Subject: Re: [U2] The math just doesn't work. And just to add to Martin's excellent answer - the normal workround for any computer system is to use integral values

Re: [U2] The math just doesn't work.

2011-09-29 Thread Mecki Foerthmann
(); } } } Result - You lie, A+B = 4203.09, C= 4203.09 -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Martin Phillips Sent: 29 September 2011 13:46 To: U2 Users List Subject: Re: [U2] The math just doesn't work

Re: [U2] The math just doesn't work.

2011-09-29 Thread Bill Haskett
- Original Message - *From:* harold.o...@clark.wa.gov *To:* U2 Users List u2-users@listserver.u2ug.org *Date:* 9/29/2011 8:40 AM *Subject:* Re: [U2] The math just doesn't work. Running Martin's code on a Universe system, got: (A+B) = C? YOU

Re: [U2] The math just doesn't work.

2011-09-29 Thread John Hester
-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Oaks, Harold Sent: Thursday, September 29, 2011 8:40 AM To: U2 Users List Subject: Re: [U2] The math just doesn't work. Running Martin's code on a Universe system, got: (A+B) = C? YOU ROCK A = 4096.9, B

Re: [U2] The math just doesn't work.

2011-09-29 Thread Oaks, Harold
to it in the Administering Universe guide. Harold -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Mecki Foerthmann Sent: Thursday, September 29, 2011 8:53 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] The math just doesn't work

Re: [U2] The math just doesn't work.

2011-09-29 Thread Kathleené M Hunter
: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Bill Haskett Sent: Thursday, September 29, 2011 9:02 AM To: U2 Users List Subject: Re: [U2] The math just doesn't work. I tried this on UD v7.2.7 on Windows 2008 R2. I added the following code

Re: [U2] The math just doesn't work.

2011-09-29 Thread Mecki Foerthmann
-Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Mecki Foerthmann Sent: Thursday, September 29, 2011 8:53 AM To: u2-users@listserver.u2ug.org Subject: Re: [U2] The math just doesn't work. You didn't read Martin's answer