On Wed, Nov 18, 2009 at 7:19 PM, Frank Warmerdam <[email protected]>wrote:
> Shaun Kolomeitz wrote: > >> Thanks Seth, >> >> It makes sense that the slowest part of the whole equation would be the >> disk operations, and there must be quite a number of disk reads/writes >> when processing imagery. Currently we use Raid Arrays that push data >> through at a rate of 300MB/s, granted if these were SSDs in Raid0 we >> could push beyond 1GB/s. Currently to process (mosaic) an 80GB image it >> takes several days to complete. This is also on 32bit hardware, and I >> suspect is only single threaded so we're limited to 3GB RAM. From what I >> understood the most optimal caching size in GDAL is 500MB, using a 512M >> window (unless that has changed). >> If you can easily lay your hands on your GSoC application than that >> would be great. We are discussing what might be possible with a very >> talented coder, who eats these types of "challenges" for breakfast ! >> Perhaps a better approach would be to use something like a grid >> computing approach then like Condor to break up the processing ? >> > > Shaun, > > I suspect that there is a gross issue with how the warping is being done, > and that it could be sped up substantially. On my two year old consumer > grade desktop I'm able to warp/reproject around 30GB/hour with gdalwarp. > > I don't want to dissuade you from investigating CUDA options, but > you might be able to get ten fold improvement by hiring an experienced > gdalwarp developer for a few hours to investigate your particular > situation. > > Best regards, > -- > > ---------------------------------------+-------------------------------------- > I set the clouds in motion - turn up | Frank Warmerdam, > [email protected] > light and sound - activate the windows | http://pobox.com/~warmerdam > and watch the world go round - Rush | Geospatial Programmer for Rent > > > _______________________________________________ > gdal-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/gdal-dev > Along these same lines, I'm looking to CUDA or Condor cluster to aid in gdal_retile. Current projects are taking from 3 weeks to 2 months to process. Will this process lend itself well to the parallel processing environment? -- Sean Villagis
_______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
