Re: [gdal-dev] Multithreading with OGR
Martin Dobias wrote: Hi Frank On Wed, Jun 23, 2010 at 4:58 PM, Frank Warmerdam warmer...@pobox.com wrote: Martin, Generally speaking it is not safe to have multiple threads making calls into a single OGRLayer at the same time. At the very least you will encounter the messing up effect on GetNextFeature(), even if you serialize calls, and if you don't serialize the calls to the OGRLayer you will also face potentially severe problems and crashes. With regard to RFC 16, only a little work in the existing code with locks was implemented. None of the TestCapability or GetLayerClone() work was done. I still think that the approach in the RFC is reasonable but I don't currently have funding to pursue the matter. There was also some pushback at the time though I don't recall all the details. My intent is to use two OGRLayer instances of the same layer: one instance for handling various stuff in main thread, the other one in worker thread for rendering. Can I expect the GetNextFeature calls to be reentrant, i.e. can they be called from both threads simultaneously using different instances of layers? Martin, Are the different OGRLayer instances coming from distinct OGRDataSource instances? If so, yes, you could normally treat them as reentrant though there might be a few special cases where that turns out not to be true. Certainly this is only a good idea in read-only activities. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Multithreading with OGR
On Fri, Jun 25, 2010 at 5:55 PM, Frank Warmerdam warmer...@pobox.com wrote: My intent is to use two OGRLayer instances of the same layer: one instance for handling various stuff in main thread, the other one in worker thread for rendering. Can I expect the GetNextFeature calls to be reentrant, i.e. can they be called from both threads simultaneously using different instances of layers? Martin, Are the different OGRLayer instances coming from distinct OGRDataSource instances? If so, yes, you could normally treat them as reentrant though there might be a few special cases where that turns out not to be true. Certainly this is only a good idea in read-only activities. Yes, I consider using distinct OGRDataSource instances for the OGRLayer instances. Could you please elaborate on the special cases you are referring to? Thanks Martin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Multithreading with OGR
Hi Frank On Wed, Jun 23, 2010 at 4:58 PM, Frank Warmerdam warmer...@pobox.com wrote: Martin, Generally speaking it is not safe to have multiple threads making calls into a single OGRLayer at the same time. At the very least you will encounter the messing up effect on GetNextFeature(), even if you serialize calls, and if you don't serialize the calls to the OGRLayer you will also face potentially severe problems and crashes. With regard to RFC 16, only a little work in the existing code with locks was implemented. None of the TestCapability or GetLayerClone() work was done. I still think that the approach in the RFC is reasonable but I don't currently have funding to pursue the matter. There was also some pushback at the time though I don't recall all the details. My intent is to use two OGRLayer instances of the same layer: one instance for handling various stuff in main thread, the other one in worker thread for rendering. Can I expect the GetNextFeature calls to be reentrant, i.e. can they be called from both threads simultaneously using different instances of layers? Regards Martin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Multithreading with OGR
Hi Ari On Wed, Jun 23, 2010 at 7:32 AM, Ari Jolma ari.jo...@gmail.com wrote: Martin, Just a comment. In Geoinformatica, which uses GTK+, I've begun to address the same kind of problem using the functions provided by GTK+. So far I've not used this on rendering and I've not thought about all the consequencies. In some expectedly lengthly GDAL operations it provides a mechanism in the function API to report progress. I've begun experimenting with this and added Gtk2-main_iteration while Gtk2-events_pending; This is something we've been using until now in QGIS - entering event loop while rendering to handle events. But this limits the overall usability a lot, because lots of GUI features have to be disabled to avoid concurrent read access and recursive redraws of map. Another limitation of that approach is the inability to make use of multi-core processors. I believe that rendering in worker threads is the way to go for desktop GIS apps. Regards Martin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Multithreading with OGR
Hi Ed On Wed, Jun 23, 2010 at 3:18 PM, Grissom, Edward (Ed) ed.gris...@intergraph.com wrote: In a previous message, Martin said: So my only problem is to avoid concurrent reads from both main thread and worker thread. How can be this best accomplished without serializing the access? Your reads are eventually going to be serialized by the disk controller anyway, so what is the problem with serializing the access ? Just do it on a small enough scale that no thread gets starved. You're right, but that is not what I had in my mind, sorry for being unclear. When rendering from worker thread, I have to avoid this kind of situations (considering one layer): 1. worker thread calls layer's ResetReading, then starts issuing GetNextFeature calls 2. before worker thread is done, main thread calls layer's ResetReading and also starts reading features 3. both main thread and worker thread do GetNextFeature calls, but both get invalid results This can happen e.g. when user tries to identify some features on map while rendering is in progress. With the serialization of access I meant serializing the calls: if main thread was going to read features, it would first wait until rendering of that layer finishes in worker thread. Clearly, with serialized access, there wouldn't be much added value when using threading. So, my main question is, if opening the same layer for worker thread is a safe operation. That is, if both main and worker thread will access (with GetNextFeature calls) their own copy of the layer, whether they both end up with correct data. Regards Martin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Multithreading with OGR
Martin Dobias wrote: So my only problem is to avoid concurrent reads from both main thread and worker thread. How can be this best accomplished without serializing the access? Your reads are eventually going to be serialized by the disk controller anyway, so what is the problem with serializing the access ? Just do it on a small enough scale that no thread gets starved. You're right, but that is not what I had in my mind, sorry for being unclear. When rendering from worker thread, I have to avoid this kind of situations (considering one layer): 1. worker thread calls layer's ResetReading, then starts issuing GetNextFeature calls 2. before worker thread is done, main thread calls layer's ResetReading and also starts reading features 3. both main thread and worker thread do GetNextFeature calls, but both get invalid results This can happen e.g. when user tries to identify some features on map while rendering is in progress. With the serialization of access I meant serializing the calls: if main thread was going to read features, it would first wait until rendering of that layer finishes in worker thread. Clearly, with serialized access, there wouldn't be much added value when using threading. So, my main question is, if opening the same layer for worker thread is a safe operation. That is, if both main and worker thread will access (with GetNextFeature calls) their own copy of the layer, whether they both end up with correct data. Martin, Generally speaking it is not safe to have multiple threads making calls into a single OGRLayer at the same time. At the very least you will encounter the messing up effect on GetNextFeature(), even if you serialize calls, and if you don't serialize the calls to the OGRLayer you will also face potentially severe problems and crashes. With regard to RFC 16, only a little work in the existing code with locks was implemented. None of the TestCapability or GetLayerClone() work was done. I still think that the approach in the RFC is reasonable but I don't currently have funding to pursue the matter. There was also some pushback at the time though I don't recall all the details. Best regards, -- ---+-- I set the clouds in motion - turn up | Frank Warmerdam, warmer...@pobox.com light and sound - activate the windows | http://pobox.com/~warmerdam and watch the world go round - Rush| Geospatial Programmer for Rent ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Multithreading with OGR
Martin, Just a comment. In Geoinformatica, which uses GTK+, I've begun to address the same kind of problem using the functions provided by GTK+. So far I've not used this on rendering and I've not thought about all the consequencies. In some expectedly lengthly GDAL operations it provides a mechanism in the function API to report progress. I've begun experimenting with this and added Gtk2-main_iteration while Gtk2-events_pending; into the progress function in addition to updating the progressbar (Geoinformatica uses GTK+ Perl, that is a line of Perl code). I assume Qt provides something similar. This makes sure the application stays responsive during the operation and allows the user to do other things. I assume I could do the same for rendering, i.e., let GTK+ process pending events occasionally. As I said, I'm not sure about all possible consequencies. Reentrancy would surely be required, but not necessarily thread-safeness, which is more demanding. Best regards, Ari Martin Dobias kirjoitti: Hi all, in my GSoC project [1] I'm trying to employ multithreading to make rendering in QGIS both faster (on multi-core machines) and more pleasant to use (not blocking GUI thread while rendering). I've been successful with making that work in ideal conditions (no concurrent reads/writes) and now the hard part comes: to make the implementation thread-safe. OGR is an important data provider, so I'd like to ask you for some hints how to do the things correctly. I've seen there have been some efforts to make OGR thread-safe - in RFC 16 [2]. In the RFC it is not clear in what parts have been implemented, but from the code I guess it ended up with only making driver registrar and data source classes thread-safe. In QGIS everything basically runs in the main thread, worker threads are spawned only when rendering the map. A worker thread for rendering only reads the features from data source and I can make sure that main thread doesn't do any write operations while the threaded rendering is in place. So my only problem is to avoid concurrent reads from both main thread and worker thread. How can be this best accomplished without serializing the access? Currently I think of creating a temporary clone of the OGRLayer for purposes of reading features in worker thread. Is that feasible - are the drivers reentrant - at least within their reading routines? Is it generally costly to create another instance of the same layer? When a write operation happens in main thread, will be the change visible in the cloned layer for worker thread or is it necessary (for some drivers) to recreate the cloned layer? [1] http://blog.qgis.org/node/144 [2] http://trac.osgeo.org/gdal/wiki/rfc16_ogr_reentrancy Thanks in advance for any hints. Regards Martin ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
