[Qgis-developer] Spatial index - what about having it automatically created and updated ?
Hi list, I have some concerns about spatial indexes in QGIS. Spatial indexes are very usefull, and for me rather mandatory as soon as you deal with more than 100 features. I would like to share some ideas about them : 1) Extend QGIS so that it can create spatial indexes for more vector format. If I am correct, for now only Shapefiles and Database vector layers can have a spatial index created by QGIS . Why not for GeoJson, CSV files, etc ? 2) If possible, create a spatial index for each vector layer added to the canvas if no one is already present. Since it is a quite fast operation, the small time needed is worth it. 3) If possible, update the spatial index (replace the old qix by a new one) every time the vector layer is updated. No need to do so for database layers, but this can be very usefull for big shapefiles. I have seen some people creating a spatial index, then as time passes, they add more and more data into the shapefile but forget to update the spatial index. Sometimes this can lead to some data not displayed in QGIS because QGIs seems to rely on the outdatedSpatial index. For me, using spatial indexes is like brushing my teeth : I do it very often without even thinking of it. But many users do not know that spatial indexes exists, or do it but forget to use them as needed. I really think QGIS should help them to improve perfs in the background. Regards Michael ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
Hi On Thu, Feb 27, 2014 at 4:47 PM, kimaidou kimai...@gmail.com wrote: Hi list, I have some concerns about spatial indexes in QGIS. Spatial indexes are very usefull, and for me rather mandatory as soon as you deal with more than 100 features. I would like to share some ideas about them : 1) Extend QGIS so that it can create spatial indexes for more vector format. If I am correct, for now only Shapefiles and Database vector layers can have a spatial index created by QGIS . Why not for GeoJson, CSV files, etc ? The create spatial index functionality is handled by data provider - in this case GDAL/OGR - at some point it was available only for shapefiles, not sure if anything changed since then. 2) If possible, create a spatial index for each vector layer added to the canvas if no one is already present. Since it is a quite fast operation, the small time needed is worth it. I am not sure this is the right thing to do it automatically. It could however suggest it for large layers for better performance. 3) If possible, update the spatial index (replace the old qix by a new one) every time the vector layer is updated. No need to do so for database layers, but this can be very usefull for big shapefiles. I have seen some people creating a spatial index, then as time passes, they add more and more data into the shapefile but forget to update the spatial index. Sometimes this can lead to some data not displayed in QGIS because QGIs seems to rely on the outdatedSpatial index. Doesn't OGR handle the update of the spatial index automatically? Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
I guess more better is to use this extremely simple batch and create all the index in one step only. sing the gdal/ogr from osgeo4w. http://trac.osgeo.org/gdal/wiki/BatchCreationIndexesForShapefilesOnDOS Regards, 2014-02-27 10:47 GMT+01:00 kimaidou kimai...@gmail.com: Hi list, I have some concerns about spatial indexes in QGIS. Spatial indexes are very usefull, and for me rather mandatory as soon as you deal with more than 100 features. I would like to share some ideas about them : 1) Extend QGIS so that it can create spatial indexes for more vector format. If I am correct, for now only Shapefiles and Database vector layers can have a spatial index created by QGIS . Why not for GeoJson, CSV files, etc ? 2) If possible, create a spatial index for each vector layer added to the canvas if no one is already present. Since it is a quite fast operation, the small time needed is worth it. 3) If possible, update the spatial index (replace the old qix by a new one) every time the vector layer is updated. No need to do so for database layers, but this can be very usefull for big shapefiles. I have seen some people creating a spatial index, then as time passes, they add more and more data into the shapefile but forget to update the spatial index. Sometimes this can lead to some data not displayed in QGIS because QGIs seems to rely on the outdatedSpatial index. For me, using spatial indexes is like brushing my teeth : I do it very often without even thinking of it. But many users do not know that spatial indexes exists, or do it but forget to use them as needed. I really think QGIS should help them to improve perfs in the background. Regards Michael ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- - Andrea Peri . . . . . . . . . qwerty àèìòù - ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
Hi 2014-02-27 11:14 GMT+01:00 Andrea Peri aperi2...@gmail.com: I guess more better is to use this extremely simple batch and create all the index in one step only. sing the gdal/ogr from osgeo4w. http://trac.osgeo.org/gdal/wiki/BatchCreationIndexesForShapefilesOnDOS Regards, As I said, I am thinking of non power users. They surely won't create nor use such a batch file, even if it is simple. I know there is also the great tool Vector / Data management tools / Create Spatial Index, which can create one for all layers in one step. But users do not use it : they do no know it exists, or they know but forget to use it. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
2014-02-27 11:08 GMT+01:00 Martin Dobias wonder...@gmail.com: Hi On Thu, Feb 27, 2014 at 4:47 PM, kimaidou kimai...@gmail.com wrote: Hi list, I have some concerns about spatial indexes in QGIS. Spatial indexes are very usefull, and for me rather mandatory as soon as you deal with more than 100 features. I would like to share some ideas about them : 1) Extend QGIS so that it can create spatial indexes for more vector format. If I am correct, for now only Shapefiles and Database vector layers can have a spatial index created by QGIS . Why not for GeoJson, CSV files, etc ? The create spatial index functionality is handled by data provider - in this case GDAL/OGR - at some point it was available only for shapefiles, not sure if anything changed since then. Ah, ok, so I should better ask in GDAL lists ! 2) If possible, create a spatial index for each vector layer added to the canvas if no one is already present. Since it is a quite fast operation, the small time needed is worth it. I am not sure this is the right thing to do it automatically. It could however suggest it for large layers for better performance. It could be an option, but I did not think it is worth it as the spatial index is fast to create. I agree we could detect if the layer is large and do it only in this case. 3) If possible, update the spatial index (replace the old qix by a new one) every time the vector layer is updated. No need to do so for database layers, but this can be very usefull for big shapefiles. I have seen some people creating a spatial index, then as time passes, they add more and more data into the shapefile but forget to update the spatial index. Sometimes this can lead to some data not displayed in QGIS because QGIs seems to rely on the outdatedSpatial index. Doesn't OGR handle the update of the spatial index automatically? I do not know, I should do some more research... This would be great indeed. Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
That's a python plugin (ftools), so it may not be available to everyone. Better to do it natively (in the data provider) doing something like OGR_DS_ExecuteSQL(CREATE SPATIAL INDEX ON tablename [DEPTH N]) - untested I think it's a great idea to do this automatically with large ogr data sources as you said you should probably ask the gdal-dev list about which drivers support spatial index creation an automatic update. cheers Etienne On Thu, Feb 27, 2014 at 7:51 AM, kimaidou kimai...@gmail.com wrote: Hi 2014-02-27 11:14 GMT+01:00 Andrea Peri aperi2...@gmail.com: I guess more better is to use this extremely simple batch and create all the index in one step only. sing the gdal/ogr from osgeo4w. http://trac.osgeo.org/gdal/wiki/BatchCreationIndexesForShapefilesOnDOS Regards, As I said, I am thinking of non power users. They surely won't create nor use such a batch file, even if it is simple. I know there is also the great tool Vector / Data management tools / Create Spatial Index, which can create one for all layers in one step. But users do not use it : they do no know it exists, or they know but forget to use it. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
On Thu, Feb 27, 2014 at 10:56 AM, Etienne Tourigny etourigny@gmail.comwrote: That's a python plugin (ftools), so it may not be available to everyone. Better to do it natively (in the data provider) doing something like OGR_DS_ExecuteSQL(CREATE SPATIAL INDEX ON tablename [DEPTH N]) - untested there is already a function QgsOgrProvider::createSpatialIndex() (for shapefiles only) which is called by QgsOgrProvider::syncToDisc() if the spatial index already exists. I think it's a great idea to do this automatically with large ogr data sources as you said you should probably ask the gdal-dev list about which drivers support spatial index creation an automatic update. cheers Etienne On Thu, Feb 27, 2014 at 7:51 AM, kimaidou kimai...@gmail.com wrote: Hi 2014-02-27 11:14 GMT+01:00 Andrea Peri aperi2...@gmail.com: I guess more better is to use this extremely simple batch and create all the index in one step only. sing the gdal/ogr from osgeo4w. http://trac.osgeo.org/gdal/wiki/BatchCreationIndexesForShapefilesOnDOS Regards, As I said, I am thinking of non power users. They surely won't create nor use such a batch file, even if it is simple. I know there is also the great tool Vector / Data management tools / Create Spatial Index, which can create one for all layers in one step. But users do not use it : they do no know it exists, or they know but forget to use it. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
Create or updte a spatial index in a table is a danger operation .. So in postgis it is availabale only to user with write grant. QGIS is unable to understand if a connection is readonly or write enable so try to do an automatic update is a trouble-matic operation if a table was open from a read-only user. Another question is why it should be complex for an user to create an index ? A second question is what kind of index ? On postgis is possible to do more than one index. A table could have more than one geometry and every one could have a spatial index or not. But an index is a space consuming and often the index is needed on 1 only geometry field menawhile the other geometry field are only for special uses. To do always automatically a spatial index on every geometry filed is really a not useful operation in these use-case. A. 2014-02-27 15:04 GMT+01:00 Etienne Tourigny etourigny@gmail.com: On Thu, Feb 27, 2014 at 10:56 AM, Etienne Tourigny etourigny@gmail.com wrote: That's a python plugin (ftools), so it may not be available to everyone. Better to do it natively (in the data provider) doing something like OGR_DS_ExecuteSQL(CREATE SPATIAL INDEX ON tablename [DEPTH N]) - untested there is already a function QgsOgrProvider::createSpatialIndex() (for shapefiles only) which is called by QgsOgrProvider::syncToDisc() if the spatial index already exists. I think it's a great idea to do this automatically with large ogr data sources as you said you should probably ask the gdal-dev list about which drivers support spatial index creation an automatic update. cheers Etienne On Thu, Feb 27, 2014 at 7:51 AM, kimaidou kimai...@gmail.com wrote: Hi 2014-02-27 11:14 GMT+01:00 Andrea Peri aperi2...@gmail.com: I guess more better is to use this extremely simple batch and create all the index in one step only. sing the gdal/ogr from osgeo4w. http://trac.osgeo.org/gdal/wiki/BatchCreationIndexesForShapefilesOnDOS Regards, As I said, I am thinking of non power users. They surely won't create nor use such a batch file, even if it is simple. I know there is also the great tool Vector / Data management tools / Create Spatial Index, which can create one for all layers in one step. But users do not use it : they do no know it exists, or they know but forget to use it. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- - Andrea Peri . . . . . . . . . qwerty àèìòù - ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Spatial index - what about having it automatically created and updated ?
Hi Etienne, On Thu, 27. Feb 2014 at 11:04:39 -0300, Etienne Tourigny wrote: there is already a function QgsOgrProvider::createSpatialIndex() (for shapefiles only) which is called by QgsOgrProvider::syncToDisc() if the spatial index already exists. And when you click on Create Spatial Index in layer properties. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de QGIS PSC member (RM) Germany IRC: jef on FreeNode -- norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH Rheinstrasse 13, 26506 Norden GF: Jelto Buurman, HR: Amtsgericht Emden, HRB 5502 ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
