C Hamilton wrote > In the processing popup, if I enter a number with less than 20 digits then > the rest of the digits become some random number. For example if I enter > > -0.1556640799496235 > > the displayed result is > > -0.15566407994962350170
AFAIK a Double-precision 64 bit floating-point (IEEE 754) can represent base 10 floating point numbers with about 15-17 significant decimal digits precision. Floating-point numbers cannot exactly represent all the real numbers in a range. -0.1556640799496235 is represented as 0xBFC3ECCCF245D4FC as 64 bit floating-point. Anyway, the "most accurate representation" of 0xBFC3ECCCF245D4FC is -0.155664079949623501697431038338 If you convert 0xBFC3ECCCF245D4FC to a base 10 real number rounding to 20 decimal digits you will display: -0.15566407994962350170 0xBFC3ECCCF245D4FB -> -0.15566407994962347 0xBFC3ECCCF245D4FC -> -0.15566407994962350 0xBFC3ECCCF245D4FD -> -0.15566407994962353 -0.15566407994962349 -> 0xBFC3ECCCF245D4FC -0.15566407994962350 -> 0xBFC3ECCCF245D4FC -0.15566407994962351 -> 0xBFC3ECCCF245D4FC See: https://en.wikipedia.org/wiki/Floating-point_arithmetic#Accuracy_problems https://en.wikipedia.org/wiki/Double-precision_floating-point_format https://www.binaryconvert.com/convert_double.html https://babbage.cs.qc.cuny.edu/IEEE-754.old/64bit.html https://www.h-schmidt.net/FloatConverter/IEEE754.html So I think this explains the issue. Regards. Andrea -- Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-f4099106.html _______________________________________________ QGIS-Developer mailing list [email protected] List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
