Hi Calvin,
That is not related to QGIS, but to the way the CPU represents floating points
numbers internally.
You can see the same behaviour in the Python console (or any Python
interpreter, and surely almost any other programming language that uses
doubles):
```
>>> v = -0.1556640799496235
>>> print(f'{v:.20f}')
-0.15566407994962350170
```
The reason those extra digits are not '0' is because doubles are not stored in
base 10, and anything beyond their standard ~16 significant digits is
approximated. For a more detailed explanation you can take a look at Python
documentation, they have a clear explanation on the topic
(https://docs.python.org/3/tutorial/floatingpoint.html).
If you really, really, really need 20 significant decimal places (that seems an
extremely rare situation, see https://xkcd.com/2170/ ) you should be looking at
your code really carefully, as you need a larger (in number of bits) than usual
floating point type; I think that neither QGIS (or standard Python or C++)
support it out of the box.
Regards,
Javier
________________________________
De: QGIS-Developer <[email protected]> en nombre de C
Hamilton <[email protected]>
Enviado: lunes, 15 de marzo de 2021 15:50:23
Para: qgis-developer
Asunto: [QGIS-Developer] Strange Behavior with QgsProcessingParameterNumber -
Bug?
In a processing algorithm I have a double QgsProcessingParameterNumber where I
have set the number of decimals metadata to be 20.
param = QgsProcessingParameterNumber(
'NUMBER',
tr('Number'),
QgsProcessingParameterNumber.Double,
defaultValue=0,
optional=False)
param.setMetadata({'widget_wrapper': { 'decimals': 20 }})
self.addParameter(param)
In the processing popup, if I enter a number with less than 20 digits then the
rest of the digits become some random number. For example if I enter
-0.1556640799496235
the displayed result is
-0.15566407994962350170
I would think that if it pads with extra digits to fill out to 20, the digits
should be 0. Have I exceeded the number of acceptable decimal digits, or is
this a bug?
Thanks,
Calvin
_______________________________________________
QGIS-Developer mailing list
[email protected]
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
