Re: [Qgis-developer] slow raster loading
Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi On Wed, Aug 10, 2011 at 12:20 AM, Peter Borissow peter.boris...@yahoo.com wrote: Also, might be a good idea to spawn a separate thread to generate the aux file in the background... It could be a possibility but the way things are set up now, stats are only gathered when really needed, and if you need then for the activity you are doing you would have to wait for them anyway. The only merrit of using a separate thread would be if we decided to start gathering complete stats on first load of a raster in the background always so that they are preemptively there when the user does something that needs stats. However I don't know if that is a good idea - if you have many rasters in a project that need stats it will spawn many threads and the IO / CPU overhead may be unpleasant. Regards Tim From: Frank Warmerdam warmer...@pobox.com To: Tim Sutton li...@linfiniti.com Cc: qgis-developer qgis-developer@lists.osgeo.org Sent: Tuesday, August 9, 2011 5:51 PM Subject: Re: [Qgis-developer] slow raster loading On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote: Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. Tim, and others, By the way, there is a mechanism in GDAL to support storing .aux.xml information in an alternative directory when the file cannot be created on the target directory due to permissions, read-only media, etc. This is enabled by setting the GDAL_PAM_PROXY_DIR option to point to a directory where files can be written. If there is a place known to QGIS that would be good for this, it might make sense to take advantage of this. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Software Developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. Regards Tim Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
[Qgis-developer] Re: GSoC Weekly report #11: DB Manager plugin for QGis
Hi all, I forgot the link. Regards. [1] http://www.qgis.org/wiki/DB_Manager_plugin_GSoC_2011 On Wed, Aug 10, 2011 at 4:01 AM, Giuseppe Sucameli brush.ty...@gmail.comwrote: Hi all, I've updated the wiki page [1] adding the weekly report of the last week. In this moment the development of the import layer feature for the posgres provider is in a pretty good state, I get no error when I compile the code :). I'm going to test it tomorrow. Regards. -- Giuseppe Sucameli -- Giuseppe Sucameli ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. But is there any downside from populating min/max on demand (e.g. like in QgsVectorDataProvider::minimumValue / maximum Value)? While it's nice to have a minimal overhead (probably buffered by the qgis server layer cache in many cases), it would be even better to have 0 overhead. Regards, Marco Am Mittwoch, 10. August 2011, 11.50:16 schrieb Tim Sutton: Hi On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. Regards Tim Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
+1 for starting now. Better to start early so we and plugin devs can start moving over, esp regarding the multi threading API, then leave it till the end. I think we could also generate some kind of report with API changes, see http://stackoverflow.com/questions/6596364/tracking-c-lib-public-api-changes - Nathan On Wed, Aug 10, 2011 at 8:32 PM, Tim Sutton li...@linfiniti.com wrote: Hi On Wed, Aug 10, 2011 at 11:58 AM, Martin Dobias wonder...@gmail.com wrote: Hi there watching the recent developments in the master branch I see various nice improvements. And they have something in common: they do _not_ break API compatibility :-) I would like to bring up this topic, because we should purge a lot of cruft and deprecated APIs on the road to 2.0. The reason why I start this is that I have been closely looking at the merge of threading branch to master. The changes in that branch are essentially of three types: 1. asynchronous map rendering (i.e. ability to browse the map while it is still being rendered) 2. thread-safe API for vector access 3. various speed optimizations Asynchronous map rendering is relatively simple to merge and requires only few API changes. But without new API for access to layers it is possible to produce crashes (e.g. map is still being rendered when you decide to start editing or do some analysis). The new API for vector access (using iterators) will require removal of the old API (select(), nextFeature()) in order to work well. And that means that virtually any functionality that uses layers' data has to be updated to new API. We need to break things in order to make threaded rendering work. The question is how to proceed and how to communicate that to users. The procedure could look like this: - set a date when we will start breaking API compatibility - create a tag in git that would be the last revision of master branch compatible with 1.x API - change plugin manager so that it allows only plugins that say qgis 2.0 is the minimal version for them to run - change plugin installer so that it works only with new plugin repository and shows plugins for qgis 2.0 Is there anything else we should consider? The point is to make the transition as smooth as possible for those using qgis-master. And what are your opinions when should we start with the breakage... in a month? in a week? tomorrow? now? :-) I think the fact that we are going to break API in 2.0 is well known and discussed often in releases, blogs, qgis hackfests etc. and is a general expectation when performing a major version numbering change. I think your approach is good (tag last api compatibility etc). As far as I am concerned we can go ahead with API breakage today. There are a number of other gut wrenching changes I would like to make for 2.0: - change theme to 'gis' theme by default - get rid of kids theme - rename default theme to 'legacy' - remove all deprecated api calls Many parts of our api have developed inconsistencies and I would like to spend some time putting my 'rpl' binary to good use trying to clean these up before we release. I think we should continue the practice of marking new api additions / changes with a @note added in 2.0 or @note replaces getLayerId in 2.0 etc. as far as possible so that people can adapt to our changes easily. Anyway +1 for API breakage from me - we will continue maintaining 1.7.x for now anyway for those looking for stability. The number of things we can reasonable backport will shrink as we mess with the API but I don't see the impact to be too big. Regards Tim Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi On Wed, Aug 10, 2011 at 12:25 PM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. But is there any downside from populating min/max on demand (e.g. like in QgsVectorDataProvider::minimumValue / maximum Value)? While it's nice to have a minimal overhead (probably buffered by the qgis server layer cache in many cases), it would be even better to have 0 overhead. Ya you make a good point. I will try update it to calculate on demand. Regards Tim Regards, Marco Am Mittwoch, 10. August 2011, 11.50:16 schrieb Tim Sutton: Hi On Wed, Aug 10, 2011 at 8:31 AM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Btw, is it possible to call computeMinMax() on demand, e.g. in QgsGDALProvider::minimumValue /maximumValue instead of the provider constructor? It probably does not matter much for the desktop experience. However, for server throughput, avoidance of opening and parsing the aux.xml file could make a difference. I think the overhead for reading the aux.xml file is really minimal and the min/max population of that if its not there is quite quick and a once off event so I wouldnt think it is going to impact the server much. I'd suggest I first run some benchmarks to see how much overhead it adds. Regards Tim Regards, Marco Am Dienstag, 9. August 2011, 21.46:45 schrieb Tim Sutton: Hi Paolo Does it do that every time you load that layer or only the first time? Does the filesystem permissions allow writing of gdal .aux.xml files? My raster stats branch merge changed some behaviours and it is probably a side effect of that. One of the changes is that gdal gets full stats when trying to determine the min max of a band. It will fetch those stats quickly from aux.xml if present of if not its going to do a full stats gather just for the basic min max stats which is where you probably experience the slowness. I will see if I can find a better way to do the basic min/max stats gathering. Regards Tim On Tue, Aug 9, 2011 at 3:49 PM, Paolo Cavallini cavall...@faunalia.it wrote: Hi all. With current master, I get bad performances in loading rasters (a 12Mb takes 5 sec); when double-clicking it from the QGIS browser, it is far slower (20 sec). From the console: Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 Any hint? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
Il 10/08/2011 12:32, Tim Sutton ha scritto: Anyway +1 for API breakage from me - we will continue maintaining 1.7.x for now anyway for those looking for stability. The number of things we can reasonable backport will shrink as we mess with the API but I don't see the impact to be too big. +1 to break API I think: - from a power user point of view (and for plugin writers), the largest impact will be in loosing (and having to rewrite) a lot of plugins, at least for several weeks or months; for this, I think it is very important to move *before breakage* all the plugins (if the authors agree) to the new infrastructure, so devs could help each other migrating to new API - it would be good to have 1.8 binaries for major distros before the breakage, so people willing to use new features *and* existing plugins will be happy. Does this sound reasonable? All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
Hi Martin, Tim I'm also in favour of breaking the API right now (it was only per coincidence no one did it so far). At the last developer meeting, someone proposed to have a list wotj the API breakages on the wiki as help for plugin authors. Regards, Marco Am Mittwoch, 10. August 2011, 12.32:32 schrieb Tim Sutton: Hi On Wed, Aug 10, 2011 at 11:58 AM, Martin Dobias wonder...@gmail.com wrote: Hi there watching the recent developments in the master branch I see various nice improvements. And they have something in common: they do _not_ break API compatibility :-) I would like to bring up this topic, because we should purge a lot of cruft and deprecated APIs on the road to 2.0. The reason why I start this is that I have been closely looking at the merge of threading branch to master. The changes in that branch are essentially of three types: 1. asynchronous map rendering (i.e. ability to browse the map while it is still being rendered) 2. thread-safe API for vector access 3. various speed optimizations Asynchronous map rendering is relatively simple to merge and requires only few API changes. But without new API for access to layers it is possible to produce crashes (e.g. map is still being rendered when you decide to start editing or do some analysis). The new API for vector access (using iterators) will require removal of the old API (select(), nextFeature()) in order to work well. And that means that virtually any functionality that uses layers' data has to be updated to new API. We need to break things in order to make threaded rendering work. The question is how to proceed and how to communicate that to users. The procedure could look like this: - set a date when we will start breaking API compatibility - create a tag in git that would be the last revision of master branch compatible with 1.x API - change plugin manager so that it allows only plugins that say qgis 2.0 is the minimal version for them to run - change plugin installer so that it works only with new plugin repository and shows plugins for qgis 2.0 Is there anything else we should consider? The point is to make the transition as smooth as possible for those using qgis-master. And what are your opinions when should we start with the breakage... in a month? in a week? tomorrow? now? :-) I think the fact that we are going to break API in 2.0 is well known and discussed often in releases, blogs, qgis hackfests etc. and is a general expectation when performing a major version numbering change. I think your approach is good (tag last api compatibility etc). As far as I am concerned we can go ahead with API breakage today. There are a number of other gut wrenching changes I would like to make for 2.0: - change theme to 'gis' theme by default - get rid of kids theme - rename default theme to 'legacy' - remove all deprecated api calls Many parts of our api have developed inconsistencies and I would like to spend some time putting my 'rpl' binary to good use trying to clean these up before we release. I think we should continue the practice of marking new api additions / changes with a @note added in 2.0 or @note replaces getLayerId in 2.0 etc. as far as possible so that people can adapt to our changes easily. Anyway +1 for API breakage from me - we will continue maintaining 1.7.x for now anyway for those looking for stability. The number of things we can reasonable backport will shrink as we mess with the API but I don't see the impact to be too big. Regards Tim Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi all, may be it is time to add to the GdalTools plugin new utility that will calculate raster statistics? Something similar to Assign projection or Build overviews. Bye -- Alexander Bruy ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Il 10/08/2011 12:49, Alexander Bruy ha scritto: may be it is time to add to the GdalTools plugin new utility that will calculate raster statistics? Something similar to Assign projection or Build overviews. Good idea; ideally this should be done in batch, also recursively, so in most situations it should be done shortly after installation. One tip should be added suggesting users to run this tool, and possibly also adding indexes to large shapefiles, to prepare all their data for fast loading. All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
In my opinion it is better to start now and have more time to update plugins and test new changes. So, +1 for breaking API now 2011/8/10 Marco Hugentobler marco.hugentob...@sourcepole.ch: Hi Martin, Tim I'm also in favour of breaking the API right now (it was only per coincidence no one did it so far). At the last developer meeting, someone proposed to have a list wotj the API breakages on the wiki as help for plugin authors. Regards, Marco Am Mittwoch, 10. August 2011, 12.32:32 schrieb Tim Sutton: Hi On Wed, Aug 10, 2011 at 11:58 AM, Martin Dobias wonder...@gmail.com wrote: Hi there watching the recent developments in the master branch I see various nice improvements. And they have something in common: they do _not_ break API compatibility :-) I would like to bring up this topic, because we should purge a lot of cruft and deprecated APIs on the road to 2.0. The reason why I start this is that I have been closely looking at the merge of threading branch to master. The changes in that branch are essentially of three types: 1. asynchronous map rendering (i.e. ability to browse the map while it is still being rendered) 2. thread-safe API for vector access 3. various speed optimizations Asynchronous map rendering is relatively simple to merge and requires only few API changes. But without new API for access to layers it is possible to produce crashes (e.g. map is still being rendered when you decide to start editing or do some analysis). The new API for vector access (using iterators) will require removal of the old API (select(), nextFeature()) in order to work well. And that means that virtually any functionality that uses layers' data has to be updated to new API. We need to break things in order to make threaded rendering work. The question is how to proceed and how to communicate that to users. The procedure could look like this: - set a date when we will start breaking API compatibility - create a tag in git that would be the last revision of master branch compatible with 1.x API - change plugin manager so that it allows only plugins that say qgis 2.0 is the minimal version for them to run - change plugin installer so that it works only with new plugin repository and shows plugins for qgis 2.0 Is there anything else we should consider? The point is to make the transition as smooth as possible for those using qgis-master. And what are your opinions when should we start with the breakage... in a month? in a week? tomorrow? now? :-) I think the fact that we are going to break API in 2.0 is well known and discussed often in releases, blogs, qgis hackfests etc. and is a general expectation when performing a major version numbering change. I think your approach is good (tag last api compatibility etc). As far as I am concerned we can go ahead with API breakage today. There are a number of other gut wrenching changes I would like to make for 2.0: - change theme to 'gis' theme by default - get rid of kids theme - rename default theme to 'legacy' - remove all deprecated api calls Many parts of our api have developed inconsistencies and I would like to spend some time putting my 'rpl' binary to good use trying to clean these up before we release. I think we should continue the practice of marking new api additions / changes with a @note added in 2.0 or @note replaces getLayerId in 2.0 etc. as far as possible so that people can adapt to our changes easily. Anyway +1 for API breakage from me - we will continue maintaining 1.7.x for now anyway for those looking for stability. The number of things we can reasonable backport will shrink as we mess with the API but I don't see the impact to be too big. Regards Tim Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Dr. Marco Hugentobler Sourcepole - Linux Open Source Solutions Churerstrasse 22, CH-8808 Pfäffikon SZ, Switzerland marco.hugentob...@sourcepole.ch http://www.sourcepole.ch Technical Advisor QGIS Project Steering Committee ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Alexander Bruy ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi Tim- I understand that the stats are only gathered when needed. Still, I think users should have the option to compute stats and possibly other support files on load. This could be accomplished via a preference/setting or a pop-up dialog. In theory, you could have a thread pool to generate the support files in the background. The size of the thread pool could be determined by a user preference/setting. Just a thought. Take Care, Peter From: Tim Sutton li...@linfiniti.com To: Peter Borissow peter.boris...@yahoo.com Cc: Frank Warmerdam warmer...@pobox.com; qgis-developer qgis-developer@lists.osgeo.org Sent: Wednesday, August 10, 2011 5:46 AM Subject: Re: [Qgis-developer] slow raster loading Hi On Wed, Aug 10, 2011 at 12:20 AM, Peter Borissow peter.boris...@yahoo.com wrote: Also, might be a good idea to spawn a separate thread to generate the aux file in the background... It could be a possibility but the way things are set up now, stats are only gathered when really needed, and if you need then for the activity you are doing you would have to wait for them anyway. The only merrit of using a separate thread would be if we decided to start gathering complete stats on first load of a raster in the background always so that they are preemptively there when the user does something that needs stats. However I don't know if that is a good idea - if you have many rasters in a project that need stats it will spawn many threads and the IO / CPU overhead may be unpleasant. Regards Tim From: Frank Warmerdam warmer...@pobox.com To: Tim Sutton li...@linfiniti.com Cc: qgis-developer qgis-developer@lists.osgeo.org Sent: Tuesday, August 9, 2011 5:51 PM Subject: Re: [Qgis-developer] slow raster loading On Tue, Aug 9, 2011 at 2:37 PM, Tim Sutton li...@linfiniti.com wrote: Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. Tim, and others, By the way, there is a mechanism in GDAL to support storing .aux.xml information in an alternative directory when the file cannot be created on the target directory due to permissions, read-only media, etc. This is enabled by setting the GDAL_PAM_PROXY_DIR option to point to a directory where files can be written. If there is a place known to QGIS that would be good for this, it might make sense to take advantage of this. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush | Geospatial Software Developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Tim Sutton - QGIS Project Steering Committee Member (Release Manager) == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Visit http://linfiniti.com to find out about: * QGIS programming and support services * Mapserver and PostGIS based hosting plans * FOSS Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net ==___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
On Wed, Aug 10, 2011 at 12:41 PM, Paolo Cavallini cavall...@faunalia.it wrote: Il 10/08/2011 12:32, Tim Sutton ha scritto: Anyway +1 for API breakage from me - we will continue maintaining 1.7.x for now anyway for those looking for stability. The number of things we can reasonable backport will shrink as we mess with the API but I don't see the impact to be too big. +1 to break API I think: - from a power user point of view (and for plugin writers), the largest impact will be in loosing (and having to rewrite) a lot of plugins, at least for several weeks or months; for this, I think it is very important to move *before breakage* all the plugins (if the authors agree) to the new infrastructure, so devs could help each other migrating to new API I do not see a reason for migrating the plugins - this should be a task of the individual authors. If an author has disappeared but the plugin is useful, someone will probably start maintaining it and upload it to the new repository. This is how it works with OSS everywhere. Let's avoid a massive import of plugins that do not work correctly due to API changes and without an active maintainer. - it would be good to have 1.8 binaries for major distros before the breakage, so people willing to use new features *and* existing plugins will be happy. We did not really plan to make a 1.8 release. I am a bit afraid that preparing another release might distract us from the road to 2.0. Anyway this is a question for the release manager. Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
Dnia środa, 10 sierpnia 2011 o 11:58:38 Martin Dobias napisał(a): - change plugin installer so that it works only with new plugin repository and shows plugins for qgis 2.0 You mean blocking all the 3rd party repositories? Or just the two central ones? ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
Il 10/08/2011 13:30, Martin Dobias ha scritto: We did not really plan to make a 1.8 release. I am a bit afraid that preparing another release might distract us from the road to 2.0. Agreed, I did not mean to have a formal release, just a quick snapshot before breakage. I'm sure someone will find it useful. All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
On Wed, Aug 10, 2011 at 1:40 PM, Borys Jurgiel li...@borysjurgiel.pl wrote: Dnia środa, 10 sierpnia 2011 o 11:58:38 Martin Dobias napisał(a): - change plugin installer so that it works only with new plugin repository and shows plugins for qgis 2.0 You mean blocking all the 3rd party repositories? Or just the two central ones? I actually meant that the button add 3rd party repositories would go away and the new repository would show up (in new installations). IIRC this is in line with what we agreed on at hackfest: to encourage plugin developers to use the new official plugin repository for all public plugins. I know we are still missing a simple tool for uploading new versions of plugins without having to upload them manually - since this was IMHO the main reason why people preferred to use their own repositories. But such a tool will not be hard to create - we only need the plugin repository to provide an API that would authorize the user and upload the plugin in one step (to avoid parsing HTML pages, doing authorization etc in several steps). Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] slow raster loading
Hi all, I had noticed the slow loading you refer to, when I got a copy of the master branch last week, but it seems to be fixed now. If you opened a raster from the command-line (i.e. qgis file.tif), the raster stats were fetched (and stored in the .aux.xml file) automatically, hence the delay. Now the proper behaviour is happening, i.e. the stats are only calculated upon demand (such as when opening the histogram tab). The Object::connect: No such signal problem is unrelated, and I have been seeing it ever since using 1.7. This should definitely get fixed... - Original Message - From: Giovanni Manghi giovanni.man...@gmail.com To: Tim Sutton li...@linfiniti.com Cc: qgis-developer qgis-developer@lists.osgeo.org Sent: Tuesday, August 9, 2011 8:57:48 PM Subject: Re: [Qgis-developer] slow raster loading Hi Tim, This may be fixed via a patch given to me by Etienne Sky - please let me know if it still does it. just tested your last commit and I still see Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /home/gio/qgis-Quantum-GIS-4b4820e/src/core/raster/qgsrasterlayer.cpp:2446 Yes thats why I was asking if the file permissions exist to write the aux.xml. If they don't behaviour should be the same as prior to my native raster stats commits i.e. stats will be recalced each time you open a project / add a layer and trigger something that needs the stats. I copied a geotiff in a folder with no write permissions and it took a lot of time to open the raster, much more than it used to take when opening the same image the first time in a folder with write permissions. Then I compiled QGIS with your last commit and now it seems it is not collecting stats/creating the aux.xml file anymore. cheers -- Giovanni -- ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS browser
I think the Browser is great, but I would suggest to have a way to update directory contents, with either a refresh button/command, or when the user opens a folder after collapsing it. regards, Etienne - Original Message - From: Paolo Cavallini cavall...@faunalia.it To: qgis-developer qgis-developer@lists.osgeo.org Cc: Sent: Monday, August 8, 2011 7:29:20 AM Subject: [Qgis-developer] QGIS browser Hi all. I'm testing the new browser; a great improvement. A few issues (I'm not sure if I'm missing something here): - drag and drop works on the legend, not on the canvas - it does not work for WMS layers - WFS are not listed Furthermore, when dragging a tiff on the legend, QGIS freezes with: self.coordXform = QgsCoordinateTransform(renderer.destinationSrs(), QgsCoordinateReferenceSystem(4326)) Warning: Object::connect: No such signal QgsGdalProvider::statusChanged( QString ) in /storage/build/Quantum-GIS/src/core/raster/qgsrasterlayer.cpp:2446 All the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] osgearth in Debian
On 08/10/2011 03:45 PM, Paolo Cavallini wrote: http://lists.alioth.debian.org/pipermail/pkg-grass-devel/2011-August/010863.html so I guess now the Debian package could include also the globe, right? thanks. exactly ciao ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] osgearth in Debian
Il 10/08/2011 18:40, Marco Bernasocchi ha scritto: exactly who can change the debian/control then? all the best. -- Paolo Cavallini: http://www.faunalia.it/pc ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
Hi. And what are your opinions when should we start with the breakage... in a month? in a week? tomorrow? now? :-) +1 for starting now. I think the Road graph plugin should be excluded from the targets of buildings when the broken API compatibility. Then, I'll post a patch and fix any problems. -- С уважением Якушев Сергей yakush...@list.ru ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] osgearth in Debian
Am Mittwoch, 10. August 2011, 15.45:14 schrieb Paolo Cavallini: http://lists.alioth.debian.org/pipermail/pkg-grass-devel/2011-August/010863 .html so I guess now the Debian package could include also the globe, right? thanks. We should wait a day or two. The Debian packages are not yet installed into the pool locations (http://incoming.debian.org/) and there are some differences to the Ubuntu packages. The Debian people wished to rename some packages. I will upload the same version as in Debian to ubuntugis tonight or tomorrow night by latest. Regards Pirmin -- Pirmin Kalberer Sourcepole - Linux Open Source Solutions http://www.sourcepole.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
On Wed, Aug 10, 2011 at 3:32 AM, Tim Sutton li...@linfiniti.com wrote:
I think the fact that we are going to break API in 2.0 is well known
and discussed often in releases, blogs, qgis hackfests etc. and is a
general expectation when performing a major version numbering change.
I think your approach is good (tag last api compatibility etc). As far
as I am concerned we can go ahead with API breakage today. There are a
number of other gut wrenching changes I would like to make for 2.0:
- change theme to 'gis' theme by default
- get rid of kids theme
- rename default theme to 'legacy'
- remove all deprecated api calls
Many parts of our api have developed inconsistencies and I would like
to spend some time putting my 'rpl' binary to good use trying to clean
these up before we release. I think we should continue the practice of
marking new api additions / changes with a @note added in 2.0 or
@note replaces getLayerId in 2.0 etc. as far as possible so that
people can adapt to our changes easily.
Anyway +1 for API breakage from me - we will continue maintaining
1.7.x for now anyway for those looking for stability. The number of
things we can reasonable backport will shrink as we mess with the API
but I don't see the impact to be too big.
Regards
Tim
Since backwards-incompatible changes are being discussed, may I also suggest
that we update PyQGIS to use version 2 of the SIP apis for QString and
QVariant?
The version 2 API is much more Pythonic and has better support for Unicode
and Python 3. It is also the only version that is compatible with PySide.
Because of this, many GUI components that use the Enthought scientific
python modules are only compatible with version 2 of the SIP APIs. Notable
projects include Matplotlib and IPython.
More information on why using API v2 is a good idea can be found here:
http://www.pyside.org/docs/pseps/psep-0101.html
Updating to the new SIP API should be a simple matter of adding:
import sip
sip.setapi('QString', 2)
sip.setapi('QVariant', 2)
To the startup routine for the Python kernel before any PyQt libraries are
added. All Python plugins that use QString or QVariant would have to replace
these with plain Python strings and dicts.
Thoughts?
-Charlie
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Breaking API and merge of threading branch
On 10/08/2011, at 19:47 , Paolo Cavallini wrote: Il 10/08/2011 13:30, Martin Dobias ha scritto: We did not really plan to make a 1.8 release. I am a bit afraid that preparing another release might distract us from the road to 2.0. Agreed, I did not mean to have a formal release, just a quick snapshot before breakage. I'm sure someone will find it useful. All the best. I have but two questions for this thread, and it's more a user question than a developer question (and if I thought like the later, I'd probably be able to answer it myself). I recall discussion of a 1.7.1 backport type release, to cover the major bugs that weren't sorted in time for the 1.7 release. Is that still on? If so, does this affect the possibility of a 1.7.1 release? Just curiosity. 1.7 is doing a good job of what I need it for - so many thanks. -ramon. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
