Markus Metz wrote: > So I misunderstood something about IEEE precision, sorry about the > confusion! Coming back to the original problem, it seems that it can > be solved for DCELL by using ".15g" instead of ".10f",
For realiable binary-decimal-binary round-trip conversions, it needs to be %.17g. DBL_EPSILON is ~2.220e-16, meaning that the next numbers after 1.0 are 1.000000000000000222, 1.000000000000000444, etc. The first few numbers after 1.0 at various precisions are: %.20g %.17g %.16g 1.0000000000000000000 1.0000000000000000 1.000000000000000 1.0000000000000002220 1.0000000000000002 1.000000000000000 1.0000000000000004441 1.0000000000000004 1.000000000000000 1.0000000000000006661 1.0000000000000007 1.000000000000001 1.0000000000000008882 1.0000000000000009 1.000000000000001 Clearly, %.16g (or less) will result in distinct binary floating point values having the same decimal representation, meaning that the round-trip conversion cannot be reliable. As the numbers get larger, so does the spacing between them (doubling at 2.0, 4.0 and 8.0). The last few before 10.0 are: %.20g %.17g %.16g 9.9999999999999911182 9.9999999999999911 9.999999999999991 9.9999999999999928946 9.9999999999999929 9.999999999999993 9.9999999999999946709 9.9999999999999947 9.999999999999995 9.9999999999999964473 9.9999999999999964 9.999999999999996 9.9999999999999982236 9.9999999999999982 9.999999999999998 So %.16g may be enough for numbers at the end of each power-of-ten range (after a few doublings but before the decimal point shifts) but not for those at the beginning. The value of DBL_DIG is 15, which indicates the the first 15 digits are always exact, while later digits may include quantisation errors. So for a decimal-binary-decimal round-trip, you can rely upon decimal values of 15 digits or less being preserved exactly. At 16 digits, double precision doesn't have sufficient accuracy to reliably preserve the 16th digit (e.g. 9.999999999999994 will come out as 9.999999999999995). -- Glynn Clements <[email protected]> _______________________________________________ grass-user mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-user
