On Thu, Oct 22, 2015 at 3:14 AM, Glynn Clements <[email protected]> wrote: > Dylan Beaudette wrote: ... > The MASK issue should have been fixed with r65591 (in trunk, not sure > if it has been backported to releasebranch_7_0 yet).
Yes, in r65764 about 4 months ago. > Ultimately, if a module uses pthreads itself, it has to ensure that > any functions which aren't thread-safe aren't called concurrently from > multiple threads. > > Accessing different maps from multiple threads is "intended" to be > safe. But the issue with the MASK was overlooked, as was the potential > for issues with GDAL-linked rasters. > > I'm starting to think that r.mapcalc's pthread support to may be more > trouble than it's worth. Unless the expression is particularly > complex, r.mapcalc tends to be I/O-bound, and pthreads doesn't really > help with that. Given all the related updated in GCC etc., perhaps it is possible to implement reasonable openMP support now? > At a minimum, I may change the thread pool code (lib/gis/worker.c) so > that the default number of workers (if WORKERS isn't set) is zero, > i.e. multiple threads will only be used if specifically requested by > the user. Maybe better since this question came up a few times. >> > But for GDAL-linked (r.external) rasters, there may be re-entrancy >> > issues with GDAL itself, meaning that even accessing different maps >> > from multiple threads is unsafe. >> >> This is troubling. Is there anyway to empirically determine what is >> safe? Last week worked on a project where I was processing daily >> rasters for a 30 year interval, involving 3 climatic variables. Due to >> the large number of files, I had to work with "external" data sources. >> For the most part, everything went as expected when the results of >> (parallel) r.mapcalc expressions were CELL maps. Not so when the >> results were FCELL or DCELL maps. > > The main problem with GDAL is that it's essentially a collection of > distinct modules in a common framework. Unless they have (and enforce) > a policy that GDALRasterIO() must be re-entrant (at least for > different maps), it may be the case that concurrent access is safe for > some formats but not others. > >> > Fixing that is potentially problematic. r.mapcalc doesn't know whether >> > its inputs are GDAL-linked (this feature is supposed to be transparent >> > to modules; there isn't a Rast_is_gdal(int fd) or similar function), >> > and lib/raster itself doesn't have any knowledge of pthreads (lib/gis >> > provides functions for managing a thread pool; r.mapcalc/r3.mapcalc >> > are currently the only modules using these functions, and r3.mapcalc >> > specifically sets WORKERS=0 because lib/raster3d isn't thread-safe). >> >> OK. Does setting "WORKERS=0" do anything when GRASS is compiled >> without pthreads? > > No. > > If you're experiencing these problems with WORKERS=0 or with GRASS > compiled without pthread support, then the problem is something else > entirely. (apparently no such issue) Markus > -- > Glynn Clements <[email protected]> > _______________________________________________ > grass-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/grass-dev _______________________________________________ grass-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-dev
