On Fri, Apr 5, 2013 at 8:55 AM, Glynn Clements <[email protected]> wrote: > > Markus Metz wrote: > >> The range of the differences between the original and the re-import is >> >> min=-6.10348170084762e-05 >> max=6.10349170528934e-05 >> >> which is magnitudes larger than the 32 bit floating point precision limit. > > No, that is the limit for values in that range. > > The range of the source data is -6.68 .. 1952.0, so the largest values > are in the range [2^10,2^11). > > Single-precision floating-point has a 24-bit significand, so values in > that range will have 13 bits to the right of the binary point, > resulting in the unit in the least place (ULP) having a magnitude of > 2^-13 ~= 1.2207-04. > > Rounding error is plus or minus half a ULP = 2^-14 ~= 6.1035-05.
I think I understand. If the largest values are in the range of 2^6 - 2^7, the ULP would have a magnitude of 2^-17 ~= 7.629395e-06? I am asking because lately I observed slightly corrupted vector coordinates in shapefiles created with recent ESRI software and suspect float-double conversion artifacts. For latlon coordinates, the largest values are in the range of 2^6 - 2^8, depending on the longitude. v.in.ogr could then, based on the largest coordinate values, suggest a threshold for snapping if topological errors are encountered. Markus M _______________________________________________ grass-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-dev
