On 2/03/2009 3:57 PM, bill lam wrote:
On Mon, 02 Mar 2009, John Machin wrote:
The question is: What is the smallest integer N such that, for each
possible IEEE 754 64-bit number, when you round it to N decimal digits
of precision for storage/transmission purposes and then convert it back
to the IEEE form, you get the original number?
The answer is: 17
Thanks John. That make sense.
BTW there is a compilation option in gnumeric "with-long-double". I'm
not sure if that use 10 byte storage or otherwise. Does it require more
than 17 digit storage format, or more importantly, compromise the
validity of numerical computation that tested for 8-byte double?
Sorry, I don't know what "with-long-double" does -- I'm just an
interested onlooker, not a dev :-)
_______________________________________________
gnumeric-list mailing list
gnumeric-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gnumeric-list
