On Mon, Oct 19, 2015 at 12:05 PM, Glynn Clements <[email protected]> wrote: > > Dylan Beaudette wrote: > >> Are there any reasons to prefer sequential operations (that do not >> alter the region) vs. parallel operations? > > Running additional jobs in parallel is only worthwhile if the > resources which they would use (CPU, memory, I/O bandwidth) would > otherwise be idle. > > Once you get to the point that a resource is saturated and jobs are > contending for it, parallel execution will be less efficient than > serial execution. > > Maybe the "parallel" command takes these factors into account > sufficiently. If it only considers CPU cores (i.e. one job per core), > you'd need to confirm that you aren't saturating I/O bandwidth or > thrashing memory or CPU caches. Try running the same sequence of tasks > with varying numbers of parallel jobs to determine the optimal value. > Needless to say, this will vary according to the nature of the task > (e.g. I/O-bound versus CPU-bound). >
Thank you Glynn, your advice confirms some empirical notes: 1. parallel processes that use data from external USB disks quickly saturate the capacity of the bus or mechanism of the drive 2. parallel processes that use data from an internal SSD can generally saturate all 8 cores of my Intel i7 My main motivation for asking this question was to determine instances where parallel operations in GRASS are _not_ safe. From my reading of the wiki, manual pages, and your recent comments on GRASS-dev, it would appear that the following operations may not be safe: 1. region-altering 2. calculations in the presence of a MASK 3. reading "external" (r.external) GDAL sources (?) 4. some mapcalc expressions In order to simplify my testing, I have disabled pthread support and invoke "parallelization" via backgrounding or GNU parallel. My examples with GNU parallel stem from the tremendous (apparent) utility of this tool, in that most "bash for loops" can be directly converted into "smart" parallel jobs. Thanks, Dylan _______________________________________________ grass-user mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-user
