2008/4/28 Yann Chemin <[EMAIL PROTECTED]>: > We have experimented a bit with parallel processing, and for a given > processor power, there is a minimum amount of operations on a pixel > that needs to be done before there is any time benefit of using > parallelization. I would also believe that multi-core parallelization > (OpenMP: easy to code in most of the case) would benefit earlier that > multi-cpus (MPICH: requires some stronger preparation about coding), > for the physical reason of transport distance of bits. > > I honestly would not know at which "level" we should integrate > parallelization capacities, it would be great indeed to have the > library understand a direct row loop and choose to split the loop by > the number of core/cpus available, if we speak about direct raster > processing. This would mean, potentially every direct raster > processing module could benefit, if they actually need it (which is > the main question, actually). > > Now many processing in raster (and vector) may now be parallelized > straight-forwardly. And this is the main thing, we often face complex > algorithms, those ones are connecting pixels together from places to > places in the map (i.e. interpolation), those required taylor-made > parallelization. Here we should need tools to ease the task of > parallelization of those complex systems. > > Finally, the codes I am working on are mostly pixel-based, either > temporal signal processing, energy-balance, adata assimilation of 1-D > vertical models, some take minutes, some hours/days, and some may go > up to a month (i dont run those last ones, I am waiting for CPU > improvement). So I would benefit directly by anything even just > running half of the raster on each core of my dual-core laptop/desktop. > > Well, I am sure there are more experienced people on this list about > parallelization, so I am waiting for comments. > > Yann > > 2008/4/28 Hamish <[EMAIL PROTECTED]>: > > > > > Yann Chemin wrote: > > > > I would like to know the planned changes for the raster library, > > > > especially the random access of pixels in the raster. > > Markus: > > > > > Not sure if all of those are actually planned, but here is a list: > > > http://grass.osgeo.org/wiki/GRASS_7_ideas_collection#Raster > > > > > > > I wanted to work on it some months back, but my daily job got more > > > > intense. > > > > In the coming future, we will need to access easily any row for > > > > parallel processing. > > > > > > One thing I wonder about for parallel processing of (serial) raster > > modules- do we really need random read access to send each individual row > > into a separate thread? The overhead with that seems counter-productive. > > Couldn't we read some GRASS_NPROC envrio variable and then split the > > overall number of rows by that number and create a small number of > > threads, ie matching the system. > > > > > > like: ceil(rows/nproc) > > nproc=4 > > rows=1035 > > chunk size=259 > > > > > > the last thread reads/processes/writes fewer rows than the others as it > > finds the EOF. > > > > > > > > another thing I still wonder about (see thread from a month or so back) > > is where to start? Modify the libs to support the concept, then tackle > > each module on their own? ie concentrate on the non-I/O limited and > > can't-do- much-about-it but throw more processor at the problem modules, > > and leave non-number crunching modules alone? -- concentrate on areas > > where we'll get the most bang for the buck / pick off low hanging fruit / > > etc? > > > > > > Hamish > > (showing off his multi-proc naivet'e) > > > > > > > > > > > ____________________________________________________________________________________ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. > http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ > > > > > > > > _______________________________________________ > > grass-dev mailing list > > [email protected] > > http://lists.osgeo.org/mailman/listinfo/grass-dev > > > > > > > > -- > Yann Chemin > International Rice Research Institute > Office: http://www.irri.org/gis > Perso: http://www.freewebs.com/ychemin >
-- Yann Chemin International Rice Research Institute Office: http://www.irri.org/gis Perso: http://www.freewebs.com/ychemin _______________________________________________ grass-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-dev
