Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin, I noticed that QGIS crashes (with the multi-threaded rendering) when you turn editing mode on for a layer if this layer exists several times in the legend (i.e. the same layer is loaded several times with different styles or whatever). I observed this at least for postgres. Cheers, Denis On 07. 01. 14 22:23, Andreas Neumann wrote: Hi Martin, I tested it and can confirm that the PostgreSQL pooling works fine and fast now. I see more than 4 Postgis connections though. But maybe these connections are from other plugins using Postgis connections as well. Good to see progress! Thanks, Andreas Am 06.01.2014 13:31, schrieb Martin Dobias: Hi Andreas On Fri, Dec 13, 2013 at 8:31 PM, Andreas Neumann a.neum...@carto.net wrote: Hi Martin, I guess I am also running into the PostgreSQL issue. In general my PostgreSQL based projects with many layers freeze after a very short time, while the SpatiaLite based projects work very nicely. Too many PostgreSQL connections? Can we limit the PostgreSQL connections or can you better re-use existing connections? A quick followup - today I have pushed some changes that introduce a connection pool for PostgreSQL. This should remove the problem with too many connections. It limits the maximum number of concurrent connections from one QGIS instance to four. Also, when the connections are not used for some time, they get closed to save resources (right now it is after one minute). Please test and let me know if things work better now. I will probably also look at pooling of SpatiaLite connections - right now the queries within SQLite are serialized, so within one SpatiaLite database you will not see any performance gain with parallel rendering. Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Denis I have just tested with one postgis layer being loaded twice. Turning editing mode on/off worked smoothly as expected. Could you please try to track down in what cases it crashes? For example: - does it happen only with postgis layers? - must be the layer loaded multiple times in order to crash? - what if rendering is not turned on when turning on editing mode? Also, a backtrace would be useful... Thanks Martin On Fri, Jan 10, 2014 at 9:34 PM, Denis Rouzaud denis.rouz...@gmail.com wrote: Hi Martin, I noticed that QGIS crashes (with the multi-threaded rendering) when you turn editing mode on for a layer if this layer exists several times in the legend (i.e. the same layer is loaded several times with different styles or whatever). I observed this at least for postgres. Cheers, Denis On 07. 01. 14 22:23, Andreas Neumann wrote: Hi Martin, I tested it and can confirm that the PostgreSQL pooling works fine and fast now. I see more than 4 Postgis connections though. But maybe these connections are from other plugins using Postgis connections as well. Good to see progress! Thanks, Andreas Am 06.01.2014 13:31, schrieb Martin Dobias: Hi Andreas On Fri, Dec 13, 2013 at 8:31 PM, Andreas Neumann a.neum...@carto.net wrote: Hi Martin, I guess I am also running into the PostgreSQL issue. In general my PostgreSQL based projects with many layers freeze after a very short time, while the SpatiaLite based projects work very nicely. Too many PostgreSQL connections? Can we limit the PostgreSQL connections or can you better re-use existing connections? A quick followup - today I have pushed some changes that introduce a connection pool for PostgreSQL. This should remove the problem with too many connections. It limits the maximum number of concurrent connections from one QGIS instance to four. Also, when the connections are not used for some time, they get closed to save resources (right now it is after one minute). Please test and let me know if things work better now. I will probably also look at pooling of SpatiaLite connections - right now the queries within SQLite are serialized, so within one SpatiaLite database you will not see any performance gain with parallel rendering. Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin, I tested it and can confirm that the PostgreSQL pooling works fine and fast now. I see more than 4 Postgis connections though. But maybe these connections are from other plugins using Postgis connections as well. Good to see progress! Thanks, Andreas Am 06.01.2014 13:31, schrieb Martin Dobias: Hi Andreas On Fri, Dec 13, 2013 at 8:31 PM, Andreas Neumann a.neum...@carto.net wrote: Hi Martin, I guess I am also running into the PostgreSQL issue. In general my PostgreSQL based projects with many layers freeze after a very short time, while the SpatiaLite based projects work very nicely. Too many PostgreSQL connections? Can we limit the PostgreSQL connections or can you better re-use existing connections? A quick followup - today I have pushed some changes that introduce a connection pool for PostgreSQL. This should remove the problem with too many connections. It limits the maximum number of concurrent connections from one QGIS instance to four. Also, when the connections are not used for some time, they get closed to save resources (right now it is after one minute). Please test and let me know if things work better now. I will probably also look at pooling of SpatiaLite connections - right now the queries within SQLite are serialized, so within one SpatiaLite database you will not see any performance gain with parallel rendering. Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin, I also confirm I don't have any more trouble with too many connections. This works like a charm Cheers, Denis On 07. 01. 14 22:23, Andreas Neumann wrote: Hi Martin, I tested it and can confirm that the PostgreSQL pooling works fine and fast now. I see more than 4 Postgis connections though. But maybe these connections are from other plugins using Postgis connections as well. Good to see progress! Thanks, Andreas Am 06.01.2014 13:31, schrieb Martin Dobias: Hi Andreas On Fri, Dec 13, 2013 at 8:31 PM, Andreas Neumann a.neum...@carto.net wrote: Hi Martin, I guess I am also running into the PostgreSQL issue. In general my PostgreSQL based projects with many layers freeze after a very short time, while the SpatiaLite based projects work very nicely. Too many PostgreSQL connections? Can we limit the PostgreSQL connections or can you better re-use existing connections? A quick followup - today I have pushed some changes that introduce a connection pool for PostgreSQL. This should remove the problem with too many connections. It limits the maximum number of concurrent connections from one QGIS instance to four. Also, when the connections are not used for some time, they get closed to save resources (right now it is after one minute). Please test and let me know if things work better now. I will probably also look at pooling of SpatiaLite connections - right now the queries within SQLite are serialized, so within one SpatiaLite database you will not see any performance gain with parallel rendering. Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Andreas On Fri, Dec 13, 2013 at 8:31 PM, Andreas Neumann a.neum...@carto.net wrote: Hi Martin, I guess I am also running into the PostgreSQL issue. In general my PostgreSQL based projects with many layers freeze after a very short time, while the SpatiaLite based projects work very nicely. Too many PostgreSQL connections? Can we limit the PostgreSQL connections or can you better re-use existing connections? A quick followup - today I have pushed some changes that introduce a connection pool for PostgreSQL. This should remove the problem with too many connections. It limits the maximum number of concurrent connections from one QGIS instance to four. Also, when the connections are not used for some time, they get closed to save resources (right now it is after one minute). Please test and let me know if things work better now. I will probably also look at pooling of SpatiaLite connections - right now the queries within SQLite are serialized, so within one SpatiaLite database you will not see any performance gain with parallel rendering. Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
I believe GeoServer (well, GeoWebCache) uses metatiling for that purpose within its WMTS/TMS. My understanding is that rather than rendering a single 256*256 pixel tile like it was asked to, it renders a grid of 4*4 (adjustable; but that's the default) of those tiles (so 1024*1024 pixels) and then clips that to get the requested tile. The result is that labels look quite good crossing tile borders. Maybe something similar could work for QGIS. Metatiling is a trick emplyed by almost every caching/tiling system. It's on the client side. While buffering is made by the server. Using both of them give the best results, but they're not alternatives. giovanni Jonathan On 12 December 2013 16:23, Bernhard Ströbl bernhard.stro...@jena.de wrote: Hi Martin, Am 12.12.2013 16:50, schrieb Martin Dobias: Hi Tim! That all sounds absolutely brilliant! Thanks for such a nice clear description of how it all fits together. I know you are only considering layer-by-layer rendering but does your design accommodate further future optimisations easily? I'm thinking of things like: * predictive / off screen rendering of 3x3 canvas dimensions after the initial render so that any pan is near instantaneous (and would trigger a new off-screen render) I have had this idea in my mind while working on the project. In theory map canvas can spawn several renderer jobs (instead of just one) and let them render just one tile of the map. Some special handling of the labeling would be necessary if we wanted labels that are allowed to cross the border of tiles. that reminds me that I needed some way to keep a stripe around the edge free of labels in QGIS server for tiling purposes (labels were cut) if you address this issue it would be nice to have this somehow customizable for QGIS server. Bernhard __ Information from ESET Mail Security, version of virus signature database 9165 (20131212) __ The message was checked by ESET Mail Security. http://www.eset.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
On Fri, Dec 13, 2013 at 2:49 PM, Marco Hugentobler marco.hugentob...@sourcepole.ch wrote: Hi Martin Wow, nice work! After first testing, it works very nice. I hope you can merge the branch quite soon. What is currently missing in order to make the merge? Hi Marco as mentioned in my earlier mail, it's mainly that not all providers have been updated yet. Then, I haven't checked how the server behaves and the point displacement renderer does not work right now. There are few other unresolved things like geometry cache for editable layer, joined layers, SLD with pixel units, split extent in renderer. Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Oh! One other thing that would be nice to keep in mind is the future support for mutliple canvases in QGIS. Regards Tim -- Tim Sutton - QGIS Project Steering Committee Member == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin, I guess I am also running into the PostgreSQL issue. In general my PostgreSQL based projects with many layers freeze after a very short time, while the SpatiaLite based projects work very nicely. Too many PostgreSQL connections? Can we limit the PostgreSQL connections or can you better re-use existing connections? Looks very promising - looking forward to testing my PostgreSQL based projects once this issue is fixed. Andreas Am 12.12.2013 16:46, schrieb Denis Rouzaud: Hi Martin, So nice to see this close to master! Just tried with a big project. At launch, I have the database login showing up (since there is too many connections apparently): And it doesn't stop showing up while in the project Will try with a smaller one ;) Cheers, Denis ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin, Just a minor thing, probably an accidental omission. Your list of data-providers that don't work doen't include the Oracle native one but I suspect probably should. From a non-dev's perspective it looks quite promising. :-) Cheers, Jonathan On 12 December 2013 12:14, Martin Dobias wonder...@gmail.com wrote: Hi everyone! [attention: long text ahead] In recent weeks I have been working on moving map rendering into background worker threads and all related infrastructure changes. There is still quite a lot of work to do, but finally I think it is a time for a preview and a broader discussion about the whole thing. Not every little QGIS feature is working yet, but things work fine with most commonly used data sources (GDAL/OGR, PostGIS, SpatiaLite). Please give it a try! The code is available in my QGIS repository on GitHub, the branch is called threading-revival: https://github.com/wonder-sk/QGIS/tree/threading-revival The plan is to continue working on the project in the following weeks to reintroduce support for features and data providers currently not supported (e.g. WMS, WFS). Hopefully by the time of feature freeze in late January the code will in condition to be merged to master, so the multi-threaded rendering can appear in QGIS 2.2 release. The project has already quite some history: it started as my GSoC project in summer of 2010, unfortunately it was not merged back to master branch because the code never get into production level quality. The scope of the project is not just about moving rendering into background: it is mostly about updating various pieces of QGIS core library and data providers to behave correctly in the case that more threads simultaneously try to access the same resource - until now the assumption always was that there was only one thread. Back in 2010, QGIS code was much less ready to change those assumptions. Now, after the release of 2.0, the code is much closer to what we need for multi-threaded rendering: both vector and raster layer code went through a major overhaul in the preparation for 2.0. What to expect from the project: 1. better user experience. Browsing the map in canvas gets much snappier - pan and zoom map smoothly with instant preview, without having to wait until rendering of the previous view is finished, without flickers or other annyoances. Even if the map takes longer to render, you are free to do any actions in the meanwhile. It is a bit hard to describe the difference of the overall feel, one needs to try it out :) 2. faster rendering of projects with more layers. Finally, it is possible to use the full power of your CPU. The rendering of map layers can be done in parallel: layers will be rendered separately at the same time and then composited together to form the final map image. In theory, rendering of two layers can get twice as fast. The speedup depends a lot on your data. 3. starting point for more asynchronous operations. With safe access to map layers from worker threads, more user actions could be processed in background without blocking GUI, e.g. opening of attribute table, running analyses, layer identification or change of selection. What not to expect from the project: - faster rendering of one individual layer. A project with one layer that took five seconds to render will still take five seconds to render. The parallelization happens at the level of map layers. With one map layer QGIS will still use just one core. Optimizing the rendering performance of an individual layer is outside of the scope of this project. What to expect from the project *right now*: things should generally work, except for the following: - data providers: delimited text, gpx, grass, mssql, sql anywhere, wfs, wms, wcs - QGIS server - point displacement renderer For testing, simply use QGIS as you would usually do and see if you feel a difference when browsing the map. In Options dialog, Rendering tab, there are few new configuration options for you to play with: 1. parallel or sequential rendering, 2. map update interval. The parallel rendering may use all your CPU power, while sequential (currently being the default) will use just one CPU core. The default map preview update interval is now set to 250ms - feel free to experiment with other values. Lower values will bring faster updates, at the expense of wasting more time doing just updates instead of real work. Parallel rendering can be switched on/off also directly in the map canvas by pressing 'P' key - useful when you want to quickly compare the difference between sequential and parallel rendering. There is another magical shortcut, 'S' key, that will show very simple stats about the rendering (currently just total rendering time), so you can again quickly compare the impact of various factors (antialiasing, parallel rendering, caching etc). These shortcuts are likely to be removed from the final version, so
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Great work Martin. I have been using this branch as my normal QGIS for the last couple of days and it feels loads better. I think the idea of the new classes and splitting the responsibly of what each one does up more. Myself, or Tamas, will update the MS SQL driver just after Christmas so that is ready to go for the merge. - Nathan On Thu, Dec 12, 2013 at 10:14 PM, Martin Dobias wonder...@gmail.com wrote: Hi everyone! [attention: long text ahead] In recent weeks I have been working on moving map rendering into background worker threads and all related infrastructure changes. There is still quite a lot of work to do, but finally I think it is a time for a preview and a broader discussion about the whole thing. Not every little QGIS feature is working yet, but things work fine with most commonly used data sources (GDAL/OGR, PostGIS, SpatiaLite). Please give it a try! The code is available in my QGIS repository on GitHub, the branch is called threading-revival: https://github.com/wonder-sk/QGIS/tree/threading-revival The plan is to continue working on the project in the following weeks to reintroduce support for features and data providers currently not supported (e.g. WMS, WFS). Hopefully by the time of feature freeze in late January the code will in condition to be merged to master, so the multi-threaded rendering can appear in QGIS 2.2 release. The project has already quite some history: it started as my GSoC project in summer of 2010, unfortunately it was not merged back to master branch because the code never get into production level quality. The scope of the project is not just about moving rendering into background: it is mostly about updating various pieces of QGIS core library and data providers to behave correctly in the case that more threads simultaneously try to access the same resource - until now the assumption always was that there was only one thread. Back in 2010, QGIS code was much less ready to change those assumptions. Now, after the release of 2.0, the code is much closer to what we need for multi-threaded rendering: both vector and raster layer code went through a major overhaul in the preparation for 2.0. What to expect from the project: 1. better user experience. Browsing the map in canvas gets much snappier - pan and zoom map smoothly with instant preview, without having to wait until rendering of the previous view is finished, without flickers or other annyoances. Even if the map takes longer to render, you are free to do any actions in the meanwhile. It is a bit hard to describe the difference of the overall feel, one needs to try it out :) 2. faster rendering of projects with more layers. Finally, it is possible to use the full power of your CPU. The rendering of map layers can be done in parallel: layers will be rendered separately at the same time and then composited together to form the final map image. In theory, rendering of two layers can get twice as fast. The speedup depends a lot on your data. 3. starting point for more asynchronous operations. With safe access to map layers from worker threads, more user actions could be processed in background without blocking GUI, e.g. opening of attribute table, running analyses, layer identification or change of selection. What not to expect from the project: - faster rendering of one individual layer. A project with one layer that took five seconds to render will still take five seconds to render. The parallelization happens at the level of map layers. With one map layer QGIS will still use just one core. Optimizing the rendering performance of an individual layer is outside of the scope of this project. What to expect from the project *right now*: things should generally work, except for the following: - data providers: delimited text, gpx, grass, mssql, sql anywhere, wfs, wms, wcs - QGIS server - point displacement renderer For testing, simply use QGIS as you would usually do and see if you feel a difference when browsing the map. In Options dialog, Rendering tab, there are few new configuration options for you to play with: 1. parallel or sequential rendering, 2. map update interval. The parallel rendering may use all your CPU power, while sequential (currently being the default) will use just one CPU core. The default map preview update interval is now set to 250ms - feel free to experiment with other values. Lower values will bring faster updates, at the expense of wasting more time doing just updates instead of real work. Parallel rendering can be switched on/off also directly in the map canvas by pressing 'P' key - useful when you want to quickly compare the difference between sequential and parallel rendering. There is another magical shortcut, 'S' key, that will show very simple stats about the rendering (currently just total rendering time), so you can again quickly compare the impact of various factors (antialiasing, parallel
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Great work Martin! I am working to speed up to vector rendering https://github.com/qgis/QGIS/pull/980 (http://hub.qgis.org/issues/8725) using current master branch as base. I wonder if I should better develop my patch using your branch as base. Anyway, cheers! De: Nathan Woodrow madman...@gmail.com Para: Martin Dobias wonder...@gmail.com CC: qgis-dev qgis-developer@lists.osgeo.org Enviado: Jueves 12 de diciembre de 2013 13:50 Asunto: Re: [Qgis-developer] QGIS Multi-threaded Rendering Great work Martin. I have been using this branch as my normal QGIS for the last couple of days and it feels loads better. I think the idea of the new classes and splitting the responsibly of what each one does up more. Myself, or Tamas, will update the MS SQL driver just after Christmas so that is ready to go for the merge. - Nathan On Thu, Dec 12, 2013 at 10:14 PM, Martin Dobias wonder...@gmail.com wrote: Hi everyone! [attention: long text ahead] In recent weeks I have been working on moving map rendering into background worker threads and all related infrastructure changes. There is still quite a lot of work to do, but finally I think it is a time for a preview and a broader discussion about the whole thing. Not every little QGIS feature is working yet, but things work fine with most commonly used data sources (GDAL/OGR, PostGIS, SpatiaLite). Please give it a try! The code is available in my QGIS repository on GitHub, the branch is called threading-revival: https://github.com/wonder-sk/QGIS/tree/threading-revival The plan is to continue working on the project in the following weeks to reintroduce support for features and data providers currently not supported (e.g. WMS, WFS). Hopefully by the time of feature freeze in late January the code will in condition to be merged to master, so the multi-threaded rendering can appear in QGIS 2.2 release. The project has already quite some history: it started as my GSoC project in summer of 2010, unfortunately it was not merged back to master branch because the code never get into production level quality. The scope of the project is not just about moving rendering into background: it is mostly about updating various pieces of QGIS core library and data providers to behave correctly in the case that more threads simultaneously try to access the same resource - until now the assumption always was that there was only one thread. Back in 2010, QGIS code was much less ready to change those assumptions. Now, after the release of 2.0, the code is much closer to what we need for multi-threaded rendering: both vector and raster layer code went through a major overhaul in the preparation for 2.0. What to expect from the project: 1. better user experience. Browsing the map in canvas gets much snappier - pan and zoom map smoothly with instant preview, without having to wait until rendering of the previous view is finished, without flickers or other annyoances. Even if the map takes longer to render, you are free to do any actions in the meanwhile. It is a bit hard to describe the difference of the overall feel, one needs to try it out :) 2. faster rendering of projects with more layers. Finally, it is possible to use the full power of your CPU. The rendering of map layers can be done in parallel: layers will be rendered separately at the same time and then composited together to form the final map image. In theory, rendering of two layers can get twice as fast. The speedup depends a lot on your data. 3. starting point for more asynchronous operations. With safe access to map layers from worker threads, more user actions could be processed in background without blocking GUI, e.g. opening of attribute table, running analyses, layer identification or change of selection. What not to expect from the project: - faster rendering of one individual layer. A project with one layer that took five seconds to render will still take five seconds to render. The parallelization happens at the level of map layers. With one map layer QGIS will still use just one core. Optimizing the rendering performance of an individual layer is outside of the scope of this project. What to expect from the project *right now*: things should generally work, except for the following: - data providers: delimited text, gpx, grass, mssql, sql anywhere, wfs, wms, wcs - QGIS server - point displacement renderer For testing, simply use QGIS as you would usually do and see if you feel a difference when browsing the map. In Options dialog, Rendering tab, there are few new configuration options for you to play with: 1. parallel or sequential rendering, 2. map update interval. The parallel rendering may use all your CPU power, while sequential (currently being the default) will use just one CPU core. The default map preview update interval is now set to 250ms - feel free to experiment with other values. Lower values will bring faster updates, at the expense
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin On Thu, Dec 12, 2013 at 2:14 PM, Martin Dobias wonder...@gmail.com wrote: Hi everyone! [attention: long text ahead] In recent weeks I have been working on moving map rendering into background worker threads and all related infrastructure changes. There is still quite a lot of work to do, but finally I think it is a time for a preview and a broader discussion about the whole thing. Not every little QGIS feature is working yet, but things work fine with most commonly used data sources (GDAL/OGR, PostGIS, SpatiaLite). Please give it a try! The code is available in my QGIS repository on GitHub, the branch is called threading-revival: https://github.com/wonder-sk/QGIS/tree/threading-revival The plan is to continue working on the project in the following weeks to reintroduce support for features and data providers currently not supported (e.g. WMS, WFS). Hopefully by the time of feature freeze in late January the code will in condition to be merged to master, so the multi-threaded rendering can appear in QGIS 2.2 release. The project has already quite some history: it started as my GSoC project in summer of 2010, unfortunately it was not merged back to master branch because the code never get into production level quality. The scope of the project is not just about moving rendering into background: it is mostly about updating various pieces of QGIS core library and data providers to behave correctly in the case that more threads simultaneously try to access the same resource - until now the assumption always was that there was only one thread. Back in 2010, QGIS code was much less ready to change those assumptions. Now, after the release of 2.0, the code is much closer to what we need for multi-threaded rendering: both vector and raster layer code went through a major overhaul in the preparation for 2.0. What to expect from the project: 1. better user experience. Browsing the map in canvas gets much snappier - pan and zoom map smoothly with instant preview, without having to wait until rendering of the previous view is finished, without flickers or other annyoances. Even if the map takes longer to render, you are free to do any actions in the meanwhile. It is a bit hard to describe the difference of the overall feel, one needs to try it out :) 2. faster rendering of projects with more layers. Finally, it is possible to use the full power of your CPU. The rendering of map layers can be done in parallel: layers will be rendered separately at the same time and then composited together to form the final map image. In theory, rendering of two layers can get twice as fast. The speedup depends a lot on your data. 3. starting point for more asynchronous operations. With safe access to map layers from worker threads, more user actions could be processed in background without blocking GUI, e.g. opening of attribute table, running analyses, layer identification or change of selection. What not to expect from the project: - faster rendering of one individual layer. A project with one layer that took five seconds to render will still take five seconds to render. The parallelization happens at the level of map layers. With one map layer QGIS will still use just one core. Optimizing the rendering performance of an individual layer is outside of the scope of this project. What to expect from the project *right now*: things should generally work, except for the following: - data providers: delimited text, gpx, grass, mssql, sql anywhere, wfs, wms, wcs - QGIS server - point displacement renderer For testing, simply use QGIS as you would usually do and see if you feel a difference when browsing the map. In Options dialog, Rendering tab, there are few new configuration options for you to play with: 1. parallel or sequential rendering, 2. map update interval. The parallel rendering may use all your CPU power, while sequential (currently being the default) will use just one CPU core. The default map preview update interval is now set to 250ms - feel free to experiment with other values. Lower values will bring faster updates, at the expense of wasting more time doing just updates instead of real work. Parallel rendering can be switched on/off also directly in the map canvas by pressing 'P' key - useful when you want to quickly compare the difference between sequential and parallel rendering. There is another magical shortcut, 'S' key, that will show very simple stats about the rendering (currently just total rendering time), so you can again quickly compare the impact of various factors (antialiasing, parallel rendering, caching etc). These shortcuts are likely to be removed from the final version, so make sure to use them while they are still there! Now, it is time for some details about the design decisions I took and their justifications. Non-developers can happily stop reading now, developers are encouraged to read that
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Even thanks for your thoughts. On Thu, Dec 12, 2013 at 8:30 PM, Even Rouault even.roua...@mines-paris.org wrote: - rasters - currently we do not have API to cancel requests for raster blocks. This means that currently we need to wait until the raster block is fully read even when we cancel the rendering job. GDAL has some support for asynchronous requests - anyone has some experience with it? Indeed there's an API in GDAL for asynchronous requests : http://trac.osgeo.org/gdal/wiki/rfc24_progressive_data_support Note that it has a real implementation only in the JPIPKAK driver (driver that implement the JPIP protocol - JPEG2000 trough the network - with the Kakadu library). For other drivers that don't have a dedicated implementation, a default implementation will just turn the async requests as a sync request (more precisely GetNextUpdatedRegion() will just do a RasterIO(), other methods mentionned in the RFC are no-op). Real cancellation / async request is generally hard to implement since the low level libraries used by drivers must support it. The analysis must be done on a per-driver basis (determine which drivers are important for you, which ones are slow, see if it can be remedied in the general synchronous case, otherwise imagine how async request could be implemented, ...) We could imagine that the default GetNextUpdatedRegion(), instead of issuing a single RasterIO() could split it up into several RasterIO() if the request window crosses several tiles (in the case of a tiled dataset) to allow cancellation between the calls to GetNextUpdatedRegion(). But that's no always a beneficial strategy. Drivers can use the fact that a RasterIO() is done on a big window to optimize things a bit, w.r.t issuing several RasterIO() on smaller windows. In GDAL 1.10, I've for example implemented multi-threaded block decoding in the OpenJPEG driver : if the window of a RasterIO() call spans over several tiles and you have several cores, then each tile will be decoded in parallel. Regarding the current API of asynchronous requests, I feel there's perhaps a lack of a way of signaling that data is ready. GetNextUpdatedRegion() works on a pulling startegy. Perhaps a callback mechanism could be usefull. That topic needs some thinking... So obviously adding async API support for other common formats would take a lot of time... I was secretly hoping that the remark in rfc24 mentioning only jpipkak driver was just a temporary state of affairs :-) However I realize that actually I do not really need to have truly asynchronous requests. The requests from QGIS are being made from within a worker thread, so it is not a problem if GDAL block that thread. The only thing I need is to be able to tell GDAL to stop reading data as soon as possible once the rendering gets cancelled. API-wise, I would like to have a method in GDALDataset, using which I could register my custom function returning cancellation status of the request (just true/false). The drivers would be responsible for calling that function whenever appropriate during RasterIO to check whether the request has been cancelled or not. It would be probably much less hassle to implement this functionality in drivers - although I understand that still many drivers using custom libraries won't be able to provide that functionality. Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Selon Martin Dobias wonder...@gmail.com: Hi Even thanks for your thoughts. On Thu, Dec 12, 2013 at 8:30 PM, Even Rouault even.roua...@mines-paris.org wrote: - rasters - currently we do not have API to cancel requests for raster blocks. This means that currently we need to wait until the raster block is fully read even when we cancel the rendering job. GDAL has some support for asynchronous requests - anyone has some experience with it? Indeed there's an API in GDAL for asynchronous requests : http://trac.osgeo.org/gdal/wiki/rfc24_progressive_data_support Note that it has a real implementation only in the JPIPKAK driver (driver that implement the JPIP protocol - JPEG2000 trough the network - with the Kakadu library). For other drivers that don't have a dedicated implementation, a default implementation will just turn the async requests as a sync request (more precisely GetNextUpdatedRegion() will just do a RasterIO(), other methods mentionned in the RFC are no-op). Real cancellation / async request is generally hard to implement since the low level libraries used by drivers must support it. The analysis must be done on a per-driver basis (determine which drivers are important for you, which ones are slow, see if it can be remedied in the general synchronous case, otherwise imagine how async request could be implemented, ...) We could imagine that the default GetNextUpdatedRegion(), instead of issuing a single RasterIO() could split it up into several RasterIO() if the request window crosses several tiles (in the case of a tiled dataset) to allow cancellation between the calls to GetNextUpdatedRegion(). But that's no always a beneficial strategy. Drivers can use the fact that a RasterIO() is done on a big window to optimize things a bit, w.r.t issuing several RasterIO() on smaller windows. In GDAL 1.10, I've for example implemented multi-threaded block decoding in the OpenJPEG driver : if the window of a RasterIO() call spans over several tiles and you have several cores, then each tile will be decoded in parallel. Regarding the current API of asynchronous requests, I feel there's perhaps a lack of a way of signaling that data is ready. GetNextUpdatedRegion() works on a pulling startegy. Perhaps a callback mechanism could be usefull. That topic needs some thinking... So obviously adding async API support for other common formats would take a lot of time... I was secretly hoping that the remark in rfc24 mentioning only jpipkak driver was just a temporary state of affairs :-) However I realize that actually I do not really need to have truly asynchronous requests. The requests from QGIS are being made from within a worker thread, so it is not a problem if GDAL block that thread. The only thing I need is to be able to tell GDAL to stop reading data as soon as possible once the rendering gets cancelled. API-wise, I would like to have a method in GDALDataset, using which I could register my custom function returning cancellation status of the request (just true/false). The drivers would be responsible for calling that function whenever appropriate during RasterIO to check whether the request has been cancelled or not. It would be probably much less hassle to implement this functionality in drivers - although I understand that still many drivers using custom libraries won't be able to provide that functionality. Yes, that's a reasonable possibility. The callback could also take as parameter a percentage of completion. That could actually be a GDALProgressFunc ( http://trac.osgeo.org/gdal/browser/trunk/gdal/port/cpl_progress.h ) Regards Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
* predictive / off screen rendering of 3x3 canvas dimensions after the initial render so that any pan is near instantaneous (and would trigger a new off-screen render) * on zoom, resample first then over render the resampled image (like open layers and other web toolkits do so you see a resampled version of the old render which gets overpainted as the new render comes in) That would be really good to keep room for this improvement. This is how maps on the web generally behave, and it's a behaviour that many of my students and customers ask about at their first impact with QGIS. giovanni == Please do not email me off-list with technical support questions. Using the lists will gain more exposure for your issues and the knowledge surrounding your issue will be shared with all. Irc: timlinux on #qgis at freenode.net == ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- Giovanni Allegri http://about.me/giovanniallegri blog: http://blog.spaziogis.it GEO+ geomatica in Italia http://bit.ly/GEOplus ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Tim! That all sounds absolutely brilliant! Thanks for such a nice clear description of how it all fits together. I know you are only considering layer-by-layer rendering but does your design accommodate further future optimisations easily? I'm thinking of things like: * predictive / off screen rendering of 3x3 canvas dimensions after the initial render so that any pan is near instantaneous (and would trigger a new off-screen render) I have had this idea in my mind while working on the project. In theory map canvas can spawn several renderer jobs (instead of just one) and let them render just one tile of the map. Some special handling of the labeling would be necessary if we wanted labels that are allowed to cross the border of tiles. * on zoom, resample first then over render the resampled image (like open layers and other web toolkits do so you see a resampled version of the old render which gets overpainted as the new render comes in) Actually that's already there :-) When you change the extent, the old map will be still present, just scaled. It will be there until the first update of the preview (now the default is set to 250ms). So, first quarter of the second you will be looking at the older resampled map (of course if your new map is rendered under 250ms, it will be shown as soon as it is done). * symbol layer render in threads - I believe even single layer draws can benefit greatly from the render-then-composite approach you are taking - rendering each feature into a buffer when symbol layers are enabled (and then compositing the buffers after rendering them) means that no feature should need to be retrieved more than once when rendering. This could be quite interesting thing to do. I guess the vector layer renderer class could actually spawn further rendering tasks for each symbol level after initial load of features and their division into groups based on their symbol. * render caching of symbol layers (so that if only one layer gets changed not all others need to be re-rendered) So you mean render caching not only the whole map layers, but also levels within layers? I am not sure how big the benefit of introducing that would be - if the added code complexity wouldn't outweigh the benefit of optimizing such case (i.e. how much of user's total time in QGIS does he/she spend waiting for update after a change to a symbol layer?) * progressive rendering (e.g. rendering in a generalised way ala A Huarte's patches and then update the display and the start a second pass render with full detail) - that way you get a very fast first render but if you stick around at that AOI the render quality improves as the second pass happens I haven't really thought about that so far. We would probably need to employ some guessing to know when does it make sense to actually do the progressive rendering and when to render just once... This 'progressive' thing reminds me of another thing worth trying: by default we could try to cache all reasonably sized vector datasets upon their load into memory. Then we would render the layers without actually querying the backend. When the rendering is done, we could then optionally run the query on backend to check whether our cache needs updating. Also, it may be interesting to build some temporary overviews for raster layers (if not present) and do the progressive rendering there - first a low quality render from the overview, then a real rendered image. I know there are possible issues with memory consumption with some of the above ideas, and they are definitely not on your current roadmap, but it would be good to at least play a little mental soccer with the above ideas and see if the architecture you have devised can accommodate such further optimisations cleanly in the future. Sure, it is always good to think ahead about possible improvements! Cheers Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin, Am 12.12.2013 16:50, schrieb Martin Dobias: Hi Tim! That all sounds absolutely brilliant! Thanks for such a nice clear description of how it all fits together. I know you are only considering layer-by-layer rendering but does your design accommodate further future optimisations easily? I'm thinking of things like: * predictive / off screen rendering of 3x3 canvas dimensions after the initial render so that any pan is near instantaneous (and would trigger a new off-screen render) I have had this idea in my mind while working on the project. In theory map canvas can spawn several renderer jobs (instead of just one) and let them render just one tile of the map. Some special handling of the labeling would be necessary if we wanted labels that are allowed to cross the border of tiles. that reminds me that I needed some way to keep a stripe around the edge free of labels in QGIS server for tiling purposes (labels were cut) if you address this issue it would be nice to have this somehow customizable for QGIS server. Bernhard __ Information from ESET Mail Security, version of virus signature database 9165 (20131212) __ The message was checked by ESET Mail Security. http://www.eset.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
I believe GeoServer (well, GeoWebCache) uses metatiling for that purpose within its WMTS/TMS. My understanding is that rather than rendering a single 256*256 pixel tile like it was asked to, it renders a grid of 4*4 (adjustable; but that's the default) of those tiles (so 1024*1024 pixels) and then clips that to get the requested tile. The result is that labels look quite good crossing tile borders. Maybe something similar could work for QGIS. Jonathan On 12 December 2013 16:23, Bernhard Ströbl bernhard.stro...@jena.de wrote: Hi Martin, Am 12.12.2013 16:50, schrieb Martin Dobias: Hi Tim! That all sounds absolutely brilliant! Thanks for such a nice clear description of how it all fits together. I know you are only considering layer-by-layer rendering but does your design accommodate further future optimisations easily? I'm thinking of things like: * predictive / off screen rendering of 3x3 canvas dimensions after the initial render so that any pan is near instantaneous (and would trigger a new off-screen render) I have had this idea in my mind while working on the project. In theory map canvas can spawn several renderer jobs (instead of just one) and let them render just one tile of the map. Some special handling of the labeling would be necessary if we wanted labels that are allowed to cross the border of tiles. that reminds me that I needed some way to keep a stripe around the edge free of labels in QGIS server for tiling purposes (labels were cut) if you address this issue it would be nice to have this somehow customizable for QGIS server. Bernhard __ Information from ESET Mail Security, version of virus signature database 9165 (20131212) __ The message was checked by ESET Mail Security. http://www.eset.com ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
And here is a video showing the difference for rendering only: http://www.lutraconsulting.co.uk/casestudies/qgis-multi-threaded-rendering Regards, Saber -Original Message- From: Martin Dobias wonder...@gmail.com To: Tim Sutton li...@linfiniti.com Cc: qgis-dev qgis-developer@lists.osgeo.org Subject: Re: [Qgis-developer] QGIS Multi-threaded Rendering Date: Thu, 12 Dec 2013 22:50:44 +0700 Hi Tim! That all sounds absolutely brilliant! Thanks for such a nice clear description of how it all fits together. I know you are only considering layer-by-layer rendering but does your design accommodate further future optimisations easily? I'm thinking of things like: * predictive / off screen rendering of 3x3 canvas dimensions after the initial render so that any pan is near instantaneous (and would trigger a new off-screen render) I have had this idea in my mind while working on the project. In theory map canvas can spawn several renderer jobs (instead of just one) and let them render just one tile of the map. Some special handling of the labeling would be necessary if we wanted labels that are allowed to cross the border of tiles. * on zoom, resample first then over render the resampled image (like open layers and other web toolkits do so you see a resampled version of the old render which gets overpainted as the new render comes in) Actually that's already there :-) When you change the extent, the old map will be still present, just scaled. It will be there until the first update of the preview (now the default is set to 250ms). So, first quarter of the second you will be looking at the older resampled map (of course if your new map is rendered under 250ms, it will be shown as soon as it is done). * symbol layer render in threads - I believe even single layer draws can benefit greatly from the render-then-composite approach you are taking - rendering each feature into a buffer when symbol layers are enabled (and then compositing the buffers after rendering them) means that no feature should need to be retrieved more than once when rendering. This could be quite interesting thing to do. I guess the vector layer renderer class could actually spawn further rendering tasks for each symbol level after initial load of features and their division into groups based on their symbol. * render caching of symbol layers (so that if only one layer gets changed not all others need to be re-rendered) So you mean render caching not only the whole map layers, but also levels within layers? I am not sure how big the benefit of introducing that would be - if the added code complexity wouldn't outweigh the benefit of optimizing such case (i.e. how much of user's total time in QGIS does he/she spend waiting for update after a change to a symbol layer?) * progressive rendering (e.g. rendering in a generalised way ala A Huarte's patches and then update the display and the start a second pass render with full detail) - that way you get a very fast first render but if you stick around at that AOI the render quality improves as the second pass happens I haven't really thought about that so far. We would probably need to employ some guessing to know when does it make sense to actually do the progressive rendering and when to render just once... This 'progressive' thing reminds me of another thing worth trying: by default we could try to cache all reasonably sized vector datasets upon their load into memory. Then we would render the layers without actually querying the backend. When the rendering is done, we could then optionally run the query on backend to check whether our cache needs updating. Also, it may be interesting to build some temporary overviews for raster layers (if not present) and do the progressive rendering there - first a low quality render from the overview, then a real rendered image. I know there are possible issues with memory consumption with some of the above ideas, and they are definitely not on your current roadmap, but it would be good to at least play a little mental soccer with the above ideas and see if the architecture you have devised can accommodate such further optimisations cleanly in the future. Sure, it is always good to think ahead about possible improvements! Cheers Martin ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer -- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete
Re: [Qgis-developer] QGIS Multi-threaded Rendering
On 12/12/2013 11:14 pm, Martin Dobias wonder...@gmail.com wrote: . - update to scale factors. I have always found the scale and raster scale factors from QgsRenderContext confusing and did not properly understand their real meaning because in various contexts (composer vs canvas) they had different meaning and value. There were also various rendering bugs due to wrong or no usage of these factors. By scaling of painter before rendering and setup of correct DPI, these factors are now always equal to one. In the future, we will be able to remove them altogether. Very excited to hear this - I'm glad I'm not the only one who found this confusing! I tested your branch last week and the improvements were dramatic. Thanks for taking the effort to tackle this. Together with all the recent render optimizations and Alvaro's work 2.2 will be a fantastic release...! Nyall ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] QGIS Multi-threaded Rendering
Hi Martin Wow, nice work! After first testing, it works very nice. I hope you can merge the branch quite soon. What is currently missing in order to make the merge? Regards, Marco On 12.12.2013 13:14, Martin Dobias wrote: Hi everyone! [attention: long text ahead] In recent weeks I have been working on moving map rendering into background worker threads and all related infrastructure changes. There is still quite a lot of work to do, but finally I think it is a time for a preview and a broader discussion about the whole thing. Not every little QGIS feature is working yet, but things work fine with most commonly used data sources (GDAL/OGR, PostGIS, SpatiaLite). Please give it a try! The code is available in my QGIS repository on GitHub, the branch is called threading-revival: https://github.com/wonder-sk/QGIS/tree/threading-revival The plan is to continue working on the project in the following weeks to reintroduce support for features and data providers currently not supported (e.g. WMS, WFS). Hopefully by the time of feature freeze in late January the code will in condition to be merged to master, so the multi-threaded rendering can appear in QGIS 2.2 release. The project has already quite some history: it started as my GSoC project in summer of 2010, unfortunately it was not merged back to master branch because the code never get into production level quality. The scope of the project is not just about moving rendering into background: it is mostly about updating various pieces of QGIS core library and data providers to behave correctly in the case that more threads simultaneously try to access the same resource - until now the assumption always was that there was only one thread. Back in 2010, QGIS code was much less ready to change those assumptions. Now, after the release of 2.0, the code is much closer to what we need for multi-threaded rendering: both vector and raster layer code went through a major overhaul in the preparation for 2.0. What to expect from the project: 1. better user experience. Browsing the map in canvas gets much snappier - pan and zoom map smoothly with instant preview, without having to wait until rendering of the previous view is finished, without flickers or other annyoances. Even if the map takes longer to render, you are free to do any actions in the meanwhile. It is a bit hard to describe the difference of the overall feel, one needs to try it out :) 2. faster rendering of projects with more layers. Finally, it is possible to use the full power of your CPU. The rendering of map layers can be done in parallel: layers will be rendered separately at the same time and then composited together to form the final map image. In theory, rendering of two layers can get twice as fast. The speedup depends a lot on your data. 3. starting point for more asynchronous operations. With safe access to map layers from worker threads, more user actions could be processed in background without blocking GUI, e.g. opening of attribute table, running analyses, layer identification or change of selection. What not to expect from the project: - faster rendering of one individual layer. A project with one layer that took five seconds to render will still take five seconds to render. The parallelization happens at the level of map layers. With one map layer QGIS will still use just one core. Optimizing the rendering performance of an individual layer is outside of the scope of this project. What to expect from the project *right now*: things should generally work, except for the following: - data providers: delimited text, gpx, grass, mssql, sql anywhere, wfs, wms, wcs - QGIS server - point displacement renderer For testing, simply use QGIS as you would usually do and see if you feel a difference when browsing the map. In Options dialog, Rendering tab, there are few new configuration options for you to play with: 1. parallel or sequential rendering, 2. map update interval. The parallel rendering may use all your CPU power, while sequential (currently being the default) will use just one CPU core. The default map preview update interval is now set to 250ms - feel free to experiment with other values. Lower values will bring faster updates, at the expense of wasting more time doing just updates instead of real work. Parallel rendering can be switched on/off also directly in the map canvas by pressing 'P' key - useful when you want to quickly compare the difference between sequential and parallel rendering. There is another magical shortcut, 'S' key, that will show very simple stats about the rendering (currently just total rendering time), so you can again quickly compare the impact of various factors (antialiasing, parallel rendering, caching etc). These shortcuts are likely to be removed from the final version, so make sure to use them while they are still there! Now, it is time for some details about the design decisions I took and their justifications. Non-developers can happily
