Thanks for the feedback Antonio and Even, it is very helpful. I probably should have specified that I want to do message passing across a cluster using MPI, rather than shared memory multithreading. I'll do some performance testing to see what the relative speeds of having workers send their data to a single writing master vs concurrent writes on NFS is. I suspect it might be highly dependant on the specific hardware involved.
-Francis On 10 February 2011 06:29, Even Rouault <[email protected]>wrote: > Le mercredi 09 février 2011 09:59:02, Antonio Valentino a écrit : > > Are you sure it is the best way to parallelize your problem? > > If you write tho the disk operation are in some way serialized. > > It is a common schema to have several worker processes/thread for CPU > > bound computations and a single IO processes/thread that collects all > > results and write them to disk. > > > > The IO process/thread int this case can also be designed to implement > > some optimization or caching mechanism that you could not apply at > > worker process level. > > > > With this schema you can bypass completely the problem of concurrent > > writes without loss of performance IMHO. > > Yes I completely agree with the approach you suggest. Genuine concurrent > writing is likely not possible on the whole chain and will result in queing > at > some point, for example at the OS level. (Unless you use some RAID, in > which > case if the file is spread over multiple disks, it is perhaps possible to > have > genuinge parallel writing of blocks of data of the same file.) > >
_______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
