Markus Metz wrote: > >> So I misunderstood something about IEEE precision, sorry about the > >> confusion! Coming back to the original problem, it seems that it can > >> be solved for DCELL by using ".15g" instead of ".10f", > > > > For realiable binary-decimal-binary round-trip conversions, it needs > > to be %.17g. > > Accordingly with single precision, for the binary-decimal-binary > round-trip it needs to be %.9g IIUC.
AFAICT, that's correct. > With regard to ticket #335 where it was decided to use %.7g and %.15g > for FCELL and DCELL, respectively, that should be changed to %.9g and > %.17g in cases where a binary-decimal-binary round-trip would take > place? Yes. > Raster values and vector attributes can be anything where very > large and very small numbers can be perfectly legal. The magnitude of the values doesn't matter, so long as you're using %g or %e rather than %f. -- Glynn Clements <[email protected]> _______________________________________________ grass-user mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-user
