Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-09 Thread Antonio Valentino
Hi Francis, Il giorno Wed, 9 Feb 2011 09:38:49 +1100 Francis Markham fmark...@gmail.com ha scritto: Is there a document anywhere specifying the best practices for parallel writes to a GDAL raster? I have an embarrasingly parallel problem that I would like to parallelise with MPI, but I'm not

Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-09 Thread Even Rouault
Le mercredi 09 février 2011 09:59:02, Antonio Valentino a écrit : Hi Francis, Il giorno Wed, 9 Feb 2011 09:38:49 +1100 Francis Markham fmark...@gmail.com ha scritto: Is there a document anywhere specifying the best practices for parallel writes to a GDAL raster? I have an embarrasingly

Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-09 Thread Francis Markham
Thanks for the feedback Antonio and Even, it is very helpful. I probably should have specified that I want to do message passing across a cluster using MPI, rather than shared memory multithreading. I'll do some performance testing to see what the relative speeds of having workers send their

[gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-08 Thread Francis Markham
Is there a document anywhere specifying the best practices for parallel writes to a GDAL raster? I have an embarrasingly parallel problem that I would like to parallelise with MPI, but I'm not sure what I am allowed to do in parallel. I would like to assign blocks exclusively to worker threads

Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-08 Thread Even Rouault
Le mardi 08 février 2011 23:38:49, Francis Markham a écrit : Is there a document anywhere specifying the best practices for parallel writes to a GDAL raster? I have an embarrasingly parallel problem that I would like to parallelise with MPI, but I'm not sure what I am allowed to do in

Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-08 Thread Francis Markham
Ouch, I guess I'll be slicing and mosaicing, sending all my data across the network, or looking for another IO library. Which approach would you recommend? Are you aware if any of the major non-GDAL libraries for popular formats (e.g. libgeotiff) support concurrent writes? -Francis On 9

Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-08 Thread Frank Warmerdam
On 11-02-08 05:58 PM, Francis Markham wrote: Ouch, I guess I'll be slicing and mosaicing, sending all my data across the network, or looking for another IO library. Which approach would you recommend? Are you aware if any of the major non-GDAL libraries for popular formats (e.g. libgeotiff)

Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-08 Thread Francis Markham
Is there any way to disable the block cache altogether? Does setting the cache size to zero work? And is there a way to do this from a python script? Cheers, Francis On 9 February 2011 10:40, Frank Warmerdam warmer...@pobox.com wrote: So, with some care you could in theory write from

Re: [gdal-dev] Best practices for concurrent writes with GDAL (was RasterIO in paralel)

2011-02-08 Thread Frank Warmerdam
On 11-02-08 08:04 PM, Francis Markham wrote: Is there any way to disable the block cache altogether? Francis, No, there is no way to disable the block cache altogether. For specific drivers you could skip past the block cache by calling GDALRasterBand::WriteBlock() but this may not work well