Re: [Qgis-user] File based Geopackage bugs
Thanks for you comment, however I did not see a reference to the issue of data loss when changing table structure (dropping columns etc). On 3/02/21 11:36 am, Nyall Dawson wrote: On Tue, 2 Feb 2021 at 12:12, Patrick Dunford wrote: I recall going back someway we were sold this Geopackage idea being implemented into Qgis because it was going to be so much better than shape files that everyone has been using up to now. I bought into these ideas by converting shape files in most of my Qgis projects into Geopackage files. However there appear to be some serious bugs in Qgis' implementation of Geopackages as a file based system, and since SQlite has an impeccable reputation, it is a reasonable conclusion that there must be significant uncorrected bugs in the Qgis code that deals with file based Geopackages. Please have a read over the threads at https://lists.osgeo.org/pipermail/qgis-developer/2020-March/060584.html and https://lists.osgeo.org/pipermail/qgis-developer/2020-May/061162.html All your issues were discussed at length in those threads -- so there's lots of background info there. Suffice to say, as you've realised, many of the issues are still outstanding and yes, geopackage can be a pain (a nightmare?) to use for some workflows. Nyall ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] File based Geopackage bugs
Hi Bernd Thanks for your comments As a previous commenter wrote, Spatialite also uses a similar format to Geopackage being also based on SQLite. So far in my testing of Spatialite I have found it to have neither issue. Spatialite converted from Geopackage creates another field called ogc_fid and ignores the fid field from the Geopackage. A test paste resulted in the ogc_fid values being automatically updated to new ones when the table is posted. I dropped several columns in a table with 38,000 records and saw no data loss with Spatialite, either. I still have to test with 2.18 and the newest master to see how they work with geopackage files before filing a bug report. I guess a question is whether the server based implementation of Geopackage has these issues. On 3/02/21 3:06 am, Bernd Vogelgesang wrote: for 2. I found a "workaround": As far as I remember, copy/paste works, but when trying to save, you get this warning/error. In the attribute table, I pick the "fid" column, enter "$id" and perform "update all". After that the layer can be saved. A novice user will never come to that solution by itself, so this a real show stopper for gpkg. ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] File based Geopackage bugs
On Tue, 2 Feb 2021 at 12:12, Patrick Dunford wrote: > > I recall going back someway we were sold this Geopackage idea being > implemented into Qgis because it was going to be so much better than > shape files that everyone has been using up to now. I bought into these > ideas by converting shape files in most of my Qgis projects into > Geopackage files. > > However there appear to be some serious bugs in Qgis' implementation of > Geopackages as a file based system, and since SQlite has an impeccable > reputation, it is a reasonable conclusion that there must be significant > uncorrected bugs in the Qgis code that deals with file based Geopackages. > Please have a read over the threads at https://lists.osgeo.org/pipermail/qgis-developer/2020-March/060584.html and https://lists.osgeo.org/pipermail/qgis-developer/2020-May/061162.html All your issues were discussed at length in those threads -- so there's lots of background info there. Suffice to say, as you've realised, many of the issues are still outstanding and yes, geopackage can be a pain (a nightmare?) to use for some workflows. Nyall > > ___ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] File based Geopackage bugs
I filed a bug report last week for the second problem, but it surely must have been previously reported. On 3/02/21 3:06 am, Bernd Vogelgesang wrote: for 2. I found a "workaround": As far as I remember, copy/paste works, but when trying to save, you get this warning/error. In the attribute table, I pick the "fid" column, enter "$id" and perform "update all". After that the layer can be saved. A novice user will never come to that solution by itself, so this a real show stopper for gpkg. Am 02.02.21 um 03:11 schrieb Patrick Dunford: I recall going back someway we were sold this Geopackage idea being implemented into Qgis because it was going to be so much better than shape files that everyone has been using up to now. I bought into these ideas by converting shape files in most of my Qgis projects into Geopackage files. However there appear to be some serious bugs in Qgis' implementation of Geopackages as a file based system, and since SQlite has an impeccable reputation, it is a reasonable conclusion that there must be significant uncorrected bugs in the Qgis code that deals with file based Geopackages. Here are two examples: 1. Changing the data table associated with a layer, specifically the operation of deleting columns, is almost 100% guaranteed to result in automatic deletion of 100% of the features in the layer when the layer is saved. I have seen this happen multiple times, whether in a layer with 92 features or one with half a million. 2. Compared to shapefiles, Geopackages add an extra field called a fid which is a unique numerical value. For new records, this value is automatically generated. This requirement of uniqueness and the field apparently being unchangeable by the end user makes it impossible to paste features from another table if the values in the fid field for pasted features happen to coincide with existing entries in the table. These two examples are making me consider changing back to shapefiles in all my projects because shapefiles did not cause either of these two problems. Supposedly a shapefile is more prone to data corruption. This may be the case but as the code in Qgis that deals with shapefiles appears to be considerably more mature, these considerations have to be reasonably weighed against each other. Particularly in the case of issue 1, this appears to be a regression since I do not recall it being an issue on Qgis 2.18 or possibly earlier versions of Qgis 3. (I use the LTS edition of Qgis) ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] File based Geopackage bugs
for 2. I found a "workaround": As far as I remember, copy/paste works, but when trying to save, you get this warning/error. In the attribute table, I pick the "fid" column, enter "$id" and perform "update all". After that the layer can be saved. A novice user will never come to that solution by itself, so this a real show stopper for gpkg. Am 02.02.21 um 03:11 schrieb Patrick Dunford: I recall going back someway we were sold this Geopackage idea being implemented into Qgis because it was going to be so much better than shape files that everyone has been using up to now. I bought into these ideas by converting shape files in most of my Qgis projects into Geopackage files. However there appear to be some serious bugs in Qgis' implementation of Geopackages as a file based system, and since SQlite has an impeccable reputation, it is a reasonable conclusion that there must be significant uncorrected bugs in the Qgis code that deals with file based Geopackages. Here are two examples: 1. Changing the data table associated with a layer, specifically the operation of deleting columns, is almost 100% guaranteed to result in automatic deletion of 100% of the features in the layer when the layer is saved. I have seen this happen multiple times, whether in a layer with 92 features or one with half a million. 2. Compared to shapefiles, Geopackages add an extra field called a fid which is a unique numerical value. For new records, this value is automatically generated. This requirement of uniqueness and the field apparently being unchangeable by the end user makes it impossible to paste features from another table if the values in the fid field for pasted features happen to coincide with existing entries in the table. These two examples are making me consider changing back to shapefiles in all my projects because shapefiles did not cause either of these two problems. Supposedly a shapefile is more prone to data corruption. This may be the case but as the code in Qgis that deals with shapefiles appears to be considerably more mature, these considerations have to be reasonably weighed against each other. Particularly in the case of issue 1, this appears to be a regression since I do not recall it being an issue on Qgis 2.18 or possibly earlier versions of Qgis 3. (I use the LTS edition of Qgis) ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] File based Geopackage bugs
Hello Patrick, Yes, there are (were?) some glitches while working with geopackages. But a lot of fixes and improvements have been made in recent releases. I remember a thread in a mailing list about the fid annoyances, but I am not sure what was the decision. Regarding geopackage vs spatialite: remember that they are different beasts. Both are independent implementations on top o SQLite. Maybe a better route would have been to make spatialite a standard, instead of creating something new, but... What version of QGIS are you using? 3.10.? Can you give it a try in more recent versions or in the nightly builds (current in development) to see if it helps Best regards, Alexandre Neto A terça, 2/02/2021, 02:13, Patrick Dunford escreveu: > I recall going back someway we were sold this Geopackage idea being > implemented into Qgis because it was going to be so much better than > shape files that everyone has been using up to now. I bought into these > ideas by converting shape files in most of my Qgis projects into > Geopackage files. > > However there appear to be some serious bugs in Qgis' implementation of > Geopackages as a file based system, and since SQlite has an impeccable > reputation, it is a reasonable conclusion that there must be significant > uncorrected bugs in the Qgis code that deals with file based Geopackages. > > Here are two examples: > > 1. Changing the data table associated with a layer, specifically the > operation of deleting columns, is almost 100% guaranteed to result in > automatic deletion of 100% of the features in the layer when the layer > is saved. I have seen this happen multiple times, whether in a layer > with 92 features or one with half a million. > > 2. Compared to shapefiles, Geopackages add an extra field called a fid > which is a unique numerical value. For new records, this value is > automatically generated. This requirement of uniqueness and the field > apparently being unchangeable by the end user makes it impossible to > paste features from another table if the values in the fid field for > pasted features happen to coincide with existing entries in the table. > > These two examples are making me consider changing back to shapefiles in > all my projects because shapefiles did not cause either of these two > problems. Supposedly a shapefile is more prone to data corruption. This > may be the case but as the code in Qgis that deals with shapefiles > appears to be considerably more mature, these considerations have to be > reasonably weighed against each other. > > Particularly in the case of issue 1, this appears to be a regression > since I do not recall it being an issue on Qgis 2.18 or possibly earlier > versions of Qgis 3. (I use the LTS edition of Qgis) > > ___ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user > ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] File based Geopackage bugs
I recall going back someway we were sold this Geopackage idea being implemented into Qgis because it was going to be so much better than shape files that everyone has been using up to now. I bought into these ideas by converting shape files in most of my Qgis projects into Geopackage files. However there appear to be some serious bugs in Qgis' implementation of Geopackages as a file based system, and since SQlite has an impeccable reputation, it is a reasonable conclusion that there must be significant uncorrected bugs in the Qgis code that deals with file based Geopackages. Here are two examples: 1. Changing the data table associated with a layer, specifically the operation of deleting columns, is almost 100% guaranteed to result in automatic deletion of 100% of the features in the layer when the layer is saved. I have seen this happen multiple times, whether in a layer with 92 features or one with half a million. 2. Compared to shapefiles, Geopackages add an extra field called a fid which is a unique numerical value. For new records, this value is automatically generated. This requirement of uniqueness and the field apparently being unchangeable by the end user makes it impossible to paste features from another table if the values in the fid field for pasted features happen to coincide with existing entries in the table. These two examples are making me consider changing back to shapefiles in all my projects because shapefiles did not cause either of these two problems. Supposedly a shapefile is more prone to data corruption. This may be the case but as the code in Qgis that deals with shapefiles appears to be considerably more mature, these considerations have to be reasonably weighed against each other. Particularly in the case of issue 1, this appears to be a regression since I do not recall it being an issue on Qgis 2.18 or possibly earlier versions of Qgis 3. (I use the LTS edition of Qgis) ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
