Markus Metz wrote:
> The range of the differences between the original and the re-import is
>
> min=-6.10348170084762e-05
> max=6.10349170528934e-05
>
> which is magnitudes larger than the 32 bit floating point precision limit.
No, that is the limit for values in that range.
The range of the source data is -6.68 .. 1952.0, so the largest values
are in the range [2^10,2^11).
Single-precision floating-point has a 24-bit significand, so values in
that range will have 13 bits to the right of the binary point,
resulting in the unit in the least place (ULP) having a magnitude of
2^-13 ~= 1.2207-04.
Rounding error is plus or minus half a ULP = 2^-14 ~= 6.1035-05.
> import numpy as np
> x = np.random.random_sample(1000000) * 1024 + 1024
> x.min(), x.max()
(1024.000187440173, 2047.9998948718983)
> y = x.astype(np.float32)
> d = y - x
> d.min(), d.max()
(-6.103512532718014e-05, 6.1035083945171209e-05)
--
Glynn Clements <[email protected]>
_______________________________________________
grass-dev mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/grass-dev
