Thanks. I had similar results with Excel and Calc. Actually got one more place of precision with Gnumeric. I was just testing one windoze spreadsheet against the others to see which one gave the best answer.
On Tue, Jul 17, 2012 at 5:37 AM, Jean Brefort <[email protected]> wrote: > It's a precision issue. Gnumeric uses C double to store real numbers and > you can't expect more that 17 decimals. Compiling with > --with-long-double would give a better precision, but this has always > been marked as experimental. > > Regards, > Jean > > Le lundi 16 juillet 2012 à 19:24 -0400, Clay Lawrence a écrit : > > Can someone else try this and see if they get the same results I do. > > I'm using Gnumeric Spreadsheet 1.10.16 > > > > > > Format a cell to 30 decimal places and enter =1/sqrt(2) > > I get 0.707106781186547460000000000000 which rounds after 17 places > > instead of 30. > > > > > > I also went to WolframAlpha and calculated 1/√2 to 30+ places; > > 0.7071067811865475244008443621048490392848359376884740 but after > > keying this in directly Gnumeric still rounds it to > > 0.707106781186547570000000000000 as soon as I hit the enter key. > > > > > > Is there a setting somewhere that I can set to get Gnumeric to use all > > of the 30 of the places it displays instead of rounding to 17 places > > and padding the remainder with zeroes? > > > > > > -- > > > > Clay > > > > "There are only 10 kinds of people in the world -- > > Those who understand binary, and those who don't." > > > > > > > > _______________________________________________ > > gnumeric-list mailing list > > [email protected] > > https://mail.gnome.org/mailman/listinfo/gnumeric-list > > > -- Clay "There are only 10 kinds of people in the world -- Those who understand binary, and those who don't."
_______________________________________________ gnumeric-list mailing list [email protected] https://mail.gnome.org/mailman/listinfo/gnumeric-list
