Re: [Qgis-developer] QgsField::length semantic
On Tue, Oct 11, 2016 at 12:40:21PM +0200, Even Rouault wrote: > Le mardi 11 octobre 2016 12:17:44, Sandro Santilli a écrit : > > Then I think the OGR provider should just avoid setting OFTReal fields > > length/precision values, upon reading (but also upon writing, as long > > as the QgsField length/precision for Double-typed fields would not be > > defined). > > > > Does it make sense ? > > On reading, should be hopefully rather harmless. Not if QgsField length/precision semantics have a defined meaning of "not-to-be-used-for-Double-types". Harmless if the defined meaning is "informative only, non-constraining". This is still an open question. > On writing, looking quickly at the provider, I couldn't see where > OGR_Fld_SetWidth()/Precision would be called with a non-zero value on a > OFTReal field if QgsField length/precision is not set There's a convertField function but indeed doesn't seem to be affecting the writing to OGR output. So this bug is fully within PostgreSQL provider making use of what should have only been informative QgsField len/precision. --strk; ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
Le mardi 11 octobre 2016 12:17:44, Sandro Santilli a écrit : > On Tue, Oct 11, 2016 at 11:53:27AM +0200, Even Rouault wrote: > > Le mardi 11 octobre 2016 11:42:12, Jürgen E. Fischer a écrit : > > > On Tue, 11. Oct 2016 at 10:45:18 +0200, Sandro Santilli wrote: > > > > Chasing a regression bug upon importing shapefile to postgresql [1] > > > > I've stumbled upon an unclear semantic of the "length" member of > > > > QgsField class [2]. > > > > > > Hm, I though that was following database semantics - at least that's > > > what I recall - maybe it was changed. > > > > > > Not sure OGR has a general semantic of it's own, maybe it follows the > > > data source and those are inconsistent. > > > > More or less. My analysis of the OGR situation at: > > http://hub.qgis.org/issues/15188#note-8 > > Then I think the OGR provider should just avoid setting OFTReal fields > length/precision values, upon reading (but also upon writing, as long > as the QgsField length/precision for Double-typed fields would not be > defined). > > Does it make sense ? On reading, should be hopefully rather harmless. On writing, looking quickly at the provider, I couldn't see where OGR_Fld_SetWidth()/Precision would be called with a non-zero value on a OFTReal field if QgsField length/precision is not set > > --strk; -- Spatialys - Geospatial professional services http://www.spatialys.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
Le mardi 11 octobre 2016 12:21:31, Sandro Santilli a écrit : > On Tue, Oct 11, 2016 at 12:11:06PM +0200, Even Rouault wrote: > > > According to this: > > > http://devzone.advantagedatabase.com/dz/webhelp/advantage9.0/server1/db > > > f_fi eld_types_and_specifications.htm > > > > > > A DBF field with extended data type "double" does not affect the > > > precision of the stored data, and the length information is simply > > > ignored. > > > > I had never seen such "double" type in use in .dbf, and it is certainly > > not handled by shapelib/OGR, and I have never seen reports that other > > shapefile producing software generate such field types. Might be a > > "Visual FoxPro" specific thing. > > So what are those OFTReal types coming from ? From DBF 'N'umeric or 'F'loat fields : https://github.com/OSGeo/gdal/blob/trunk/gdal/ogr/ogrsf_frmts/shape/dbfopen.c#L1273 which are mapped to Shapelib FTDouble which is then mapped to OGR OFTReal -- Spatialys - Geospatial professional services http://www.spatialys.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
On Tue, Oct 11, 2016 at 12:11:06PM +0200, Even Rouault wrote: > > According to this: > > http://devzone.advantagedatabase.com/dz/webhelp/advantage9.0/server1/dbf_fi > > eld_types_and_specifications.htm > > > > A DBF field with extended data type "double" does not affect the > > precision of the stored data, and the length information is simply > > ignored. > > I had never seen such "double" type in use in .dbf, and it is certainly not > handled by shapelib/OGR, and I have never seen reports that other shapefile > producing software generate such field types. Might be a "Visual FoxPro" > specific thing. So what are those OFTReal types coming from ? According to shapelib, the test shapefile field type is: Field shape_area is an FTDouble with width 19 and precision 11 > > After all, a "Double" in a DBF is just 8 bytes anyway, so why bother > > attempting to go beyond that ? Qgis is storing values in a Double too, > > and the only reason I see to support "numeric" is storing arbitrary > > precision values, which don't seem to be supported by OGR: > > http://www.gdal.org/ogr__core_8h.html#a787194bea637faf12d61643124a7c9fc > > True, OGR ends up storing numeric fields as C double. The width/precision is > mostly informative (and occasionnaly indeed an annoyance when backends start > using it...) Ok so my proposed fix for issue 15188 is simply to revert the PR, which is the one making (mis)use of that information while writing in the database. --strk; ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
On Tue, Oct 11, 2016 at 11:53:27AM +0200, Even Rouault wrote: > Le mardi 11 octobre 2016 11:42:12, Jürgen E. Fischer a écrit : > > On Tue, 11. Oct 2016 at 10:45:18 +0200, Sandro Santilli wrote: > > > Chasing a regression bug upon importing shapefile to postgresql [1] > > > I've stumbled upon an unclear semantic of the "length" member of > > > QgsField class [2]. > > > > Hm, I though that was following database semantics - at least that's what I > > recall - maybe it was changed. > > > > Not sure OGR has a general semantic of it's own, maybe it follows the data > > source and those are inconsistent. > > More or less. My analysis of the OGR situation at: > http://hub.qgis.org/issues/15188#note-8 Then I think the OGR provider should just avoid setting OFTReal fields length/precision values, upon reading (but also upon writing, as long as the QgsField length/precision for Double-typed fields would not be defined). Does it make sense ? --strk; ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
> According to this: > http://devzone.advantagedatabase.com/dz/webhelp/advantage9.0/server1/dbf_fi > eld_types_and_specifications.htm > > A DBF field with extended data type "double" does not affect the > precision of the stored data, and the length information is simply > ignored. I had never seen such "double" type in use in .dbf, and it is certainly not handled by shapelib/OGR, and I have never seen reports that other shapefile producing software generate such field types. Might be a "Visual FoxPro" specific thing. > > After all, a "Double" in a DBF is just 8 bytes anyway, so why bother > attempting to go beyond that ? Qgis is storing values in a Double too, > and the only reason I see to support "numeric" is storing arbitrary > precision values, which don't seem to be supported by OGR: > http://www.gdal.org/ogr__core_8h.html#a787194bea637faf12d61643124a7c9fc True, OGR ends up storing numeric fields as C double. The width/precision is mostly informative (and occasionnaly indeed an annoyance when backends start using it...) > > --strk; > ___ > Qgis-developer mailing list > Qgis-developer@lists.osgeo.org > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Spatialys - Geospatial professional services http://www.spatialys.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
On Tue, Oct 11, 2016 at 11:42:12AM +0200, Jürgen E. Fischer wrote: > On Tue, 11. Oct 2016 at 10:45:18 +0200, Sandro Santilli wrote: > > Chasing a regression bug upon importing shapefile to postgresql [1] > > I've stumbled upon an unclear semantic of the "length" member of > > QgsField class [2]. > > Hm, I though that was following database semantics - at least that's what I > recall - maybe it was changed. By "database semantics" you mean PostgreSQL "numeric" semantic ? PostgreSQL numeric has "precision" and "scale", where "precision" is total number of digits (excluding comma) and "scale" is max number of decimal digits (right of the comma, truncated as needed). > Not sure OGR has a general semantic of it's own, maybe it follows the data > source and those are inconsistent. OGR provider decrements the length by one if precision is non-zero, upon reading: https://github.com/qgis/QGIS/blob/master/src/providers/ogr/qgsogrprovider.cpp#L885 And re-adds one upon writing (converting): https://github.com/qgis/QGIS/blob/master/src/providers/ogr/qgsogrprovider.cpp#L94 In both cases it does so by just looking at the presence of a non-zero precision, regardless of datatype. > For what I know in shapes and postgres the semantics is different as the > decimal separator is considered part of the precision in one, but not the > other. According to this: http://devzone.advantagedatabase.com/dz/webhelp/advantage9.0/server1/dbf_field_types_and_specifications.htm A DBF field with extended data type "double" does not affect the precision of the stored data, and the length information is simply ignored. After all, a "Double" in a DBF is just 8 bytes anyway, so why bother attempting to go beyond that ? Qgis is storing values in a Double too, and the only reason I see to support "numeric" is storing arbitrary precision values, which don't seem to be supported by OGR: http://www.gdal.org/ogr__core_8h.html#a787194bea637faf12d61643124a7c9fc --strk; ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
Le mardi 11 octobre 2016 11:42:12, Jürgen E. Fischer a écrit : > On Tue, 11. Oct 2016 at 10:45:18 +0200, Sandro Santilli wrote: > > Chasing a regression bug upon importing shapefile to postgresql [1] > > I've stumbled upon an unclear semantic of the "length" member of > > QgsField class [2]. > > Hm, I though that was following database semantics - at least that's what I > recall - maybe it was changed. > > Not sure OGR has a general semantic of it's own, maybe it follows the data > source and those are inconsistent. More or less. My analysis of the OGR situation at: http://hub.qgis.org/issues/15188#note-8 > > For what I know in shapes and postgres the semantics is different as the > decimal separator is considered part of the precision in one, but not the > other. > > > Jürgen -- Spatialys - Geospatial professional services http://www.spatialys.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
On Tue, 11. Oct 2016 at 10:45:18 +0200, Sandro Santilli wrote: > Chasing a regression bug upon importing shapefile to postgresql [1] > I've stumbled upon an unclear semantic of the "length" member of > QgsField class [2]. Hm, I though that was following database semantics - at least that's what I recall - maybe it was changed. Not sure OGR has a general semantic of it's own, maybe it follows the data source and those are inconsistent. For what I know in shapes and postgres the semantics is different as the decimal separator is considered part of the precision in one, but not the other. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de QGIS release manager (PSC) GermanyIRC: jef on FreeNode pgpTMaPhDfD7U.pgp Description: PGP signature ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
Adding Arnaud to the loop, being the author of the PR which added constrained numeric fields in PostgreSQL provider [1] [1] https://github.com/qgis/qgis/commit/92f71b696ca93c792ae5602ed82863fcef0e5006 Arnaud: what was the reason for constraining that field ? The semantic of QgsField's length and precision attributes aren't fully defined and they seem not to match PostgreSQL numeric relatives "precision" and "scale" [2] [2] https://www.postgresql.org/docs/9.5/static/datatype-numeric.html#DATATYPE-NUMERIC-DECIMAL --strk; On Tue, Oct 11, 2016 at 10:51:34AM +0200, Sandro Santilli wrote: > On Tue, Oct 11, 2016 at 10:47:50AM +0200, Paolo Cavallini wrote: > > Il 11 ottobre 2016 10:45:18 CEST, Sandro Santilli ha scritto: > > > >Does anyone have an idea about how should QgsField::length be > > >interpreted ? > > > > Isn't it related to the comma being counted in real fields? > > The bug ? Probably. QgsOgrProvider::convertField adds one byte > for the comma while QgsPostgresProvider::convertField does not > > But what I'd like to know is whether or not QgsField::length > is supposed to count or not the comma (or what else is meant > to be expressing) for a QVariant::Double and for other types. > > --strk; > ___ > Qgis-developer mailing list > Qgis-developer@lists.osgeo.org > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
On Tue, Oct 11, 2016 at 10:47:50AM +0200, Paolo Cavallini wrote: > Il 11 ottobre 2016 10:45:18 CEST, Sandro Santilli ha scritto: > >Does anyone have an idea about how should QgsField::length be > >interpreted ? > > Isn't it related to the comma being counted in real fields? The bug ? Probably. QgsOgrProvider::convertField adds one byte for the comma while QgsPostgresProvider::convertField does not But what I'd like to know is whether or not QgsField::length is supposed to count or not the comma (or what else is meant to be expressing) for a QVariant::Double and for other types. --strk; ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QgsField::length semantic
Il 11 ottobre 2016 10:45:18 CEST, Sandro Santilli ha scritto: >Chasing a regression bug upon importing shapefile to postgresql [1] >I've stumbled upon an unclear semantic of the "length" member of >QgsField class [2]. > >The Ogr provider, upon loading shapefiles, uses the DBF field width >_minus_one_, for reasons that are unknown to me. Upon writing a layer >to PostgreSQL, the missing _one_ results in values being too large >for the field size. > >So the solution to this bug seems to be a clear definition of what >QgsField::length is supposed to be, beacuse either Ogr provider is >doing a mistake in reducing that field by 1 or PostgreSQL provider >is doing a mistake in NOT augmenting that field by 1. > >Does anyone have an idea about how should QgsField::length be >interpreted ? > > >[1] http://hub.qgis.org/issues/15188 >[2] >https://qgis.org/api/classQgsField.html#a8490cd90a5089a7f04bb97c725e54be9 > >--strk; > > () Free GIS & Flash consultant/developer > /\ https://strk.kbt.io/services.html >___ >Qgis-developer mailing list >Qgis-developer@lists.osgeo.org >List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer >Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer Isn't it related to the comma being counted in real fields? All the best. -- Sent from mobile. Sorry for being short___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
