I figured that something like that was probably causing this. No, I don't need 20 digits. i was seeing data with 16 digits from KMLs.
Thanks, Calvin On Mon, Mar 15, 2021 at 11:54 AM javier <[email protected]> wrote: > Hi Calvin, > > > That is not related to QGIS, but to the way the CPU represents floating > points numbers internally. > > You can see the same behaviour in the Python console (or any Python > interpreter, and surely almost any other programming language that > uses doubles): > > ``` > > >>> v = -0.1556640799496235 > > >>> print(f'{v:.20f}') > > -0.15566407994962350170 > > ``` > > The reason those extra digits are not '0' is because doubles are not > stored in base 10, and anything beyond their standard ~16 significant > digits is approximated. For a more detailed explanation you can take a look > at Python documentation, they have a clear explanation on the topic ( > https://docs.python.org/3/tutorial/floatingpoint.html). > > > > If you really, really, really need 20 significant decimal places (that > seems an extremely rare situation, see https://xkcd.com/2170/ ) you > should be looking at your code really carefully, as you need a larger (in > number of bits) than usual floating point type; I think that neither QGIS > (or standard Python or C++) support it out of the box. > > > Regards, > > Javier > ------------------------------ > *De:* QGIS-Developer <[email protected]> en nombre > de C Hamilton <[email protected]> > *Enviado:* lunes, 15 de marzo de 2021 15:50:23 > *Para:* qgis-developer > *Asunto:* [QGIS-Developer] Strange Behavior with > QgsProcessingParameterNumber - Bug? > > In a processing algorithm I have a double QgsProcessingParameterNumber > where I have set the number of decimals metadata to be 20. > > param = QgsProcessingParameterNumber( > 'NUMBER', > tr('Number'), > QgsProcessingParameterNumber.Double, > defaultValue=0, > optional=False) > param.setMetadata({'widget_wrapper': { 'decimals': 20 }}) > self.addParameter(param) > > In the processing popup, if I enter a number with less than 20 digits then > the rest of the digits become some random number. For example if I enter > > -0.1556640799496235 > > the displayed result is > > -0.15566407994962350170 > > I would think that if it pads with extra digits to fill out to 20, the > digits should be 0. Have I exceeded the number of acceptable decimal > digits, or is this a bug? > > Thanks, > > Calvin >
_______________________________________________ QGIS-Developer mailing list [email protected] List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
