It is not nice (to yourself) to claim there is a bug before further examination or reflection :-)
Fact 0: 64-bit IEEE floating point has 53 bits or about 16 decimal digits of precision. Fact 1: By default J uses fixed precision numbers: 1 byte booleans, 32 bit integers (64 bit on 64 bit CPUs), 64 bit floating point numbers, 128 bit complex numbers. This is partly historical but mainly for efficiency reasons: fixed precision numbers (machine numbers) are one to two orders of magnitude faster than extended precision numbers. Fact 2: %:2 is a 64-bit number. Fact 3: x: y converts y to an extended precision integer or a rational number. In the case of %:2, since it is not an integer, it converts it to a rational. Fact 4: x: can not manufacture digits that are not there. You may think of %:2 as the square root of 2, but the interpreter doesn't know about that. As far as it is concerned %:2 is just another 64-bit floating point number. Fact 5: Like other primitives, x: uses a tolerance for comparisons. This tolerance is normally 2^_44 but can be overridden using the !. conjunction, x:!.e y . Thus: 0j20 ": %: 2 1.41421356237309510000 0j20 ": x: %: 2 NB. default tolerance of 2^_44 1.41421356237313909704 0j20 ": x:!.0 %: 2 NB. 0 tolerance 1.41421356237309536752 Why not use 0 tolerance all the time? Consider: x:!.0 ]0.071 170536305889763r2401919801264265 x: 0.071 71r1000 Which is the better answer? (Yes, this is a trick question.) ----- Original Message ----- From: "p j" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Tuesday, February 21, 2006 11:47 AM Subject: [Jprogramming] re: Numeric precision bug? Roger, Ok, the formula produces garbage for very small k... any fibonacci number smaller than 10^10. It also doesnt include the last binet term that is only useful for determining small fibonacci numbers. Anyways, unrelated to original question: This seems like a real precision bug though... 102j100 ": x:(%: 2) 1.4142135623731390970412181031989721904863670411925480575736180767789366814500092718645843712378263880 amount cut off to match below: 1.4142135623731390970412181031989 windows calculator: for sqrt 2 1.4142135623730950488016887242097 the 2 numbers diverge around decimal position 12 ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
