Hello John, I developed the Parallel GDAL2Tiles application (info at http://www.maptiler.com/) and I am using the ParallelPython module for GDAL automation on a computer cluster - have a look at http://www.parallelpython.com/.
It runs automatically on SMP and multi-core computers and it is simple to setup in a cluster mode as well. Parallel programming for simultaneously running python processes with this module is quite straight forward. I can just recommend that - it runs well in several production environments, including Amazon EC2 cluster (I made some patches for that). I was also testing the light-weight open-source implementation of Google's MapReduce from Nokia research lab available at http://discoproject.org/. This is also usable for GDAL processing of huge datasets. I would like to present the MapTiler at FOSS4G in Sydney in case I get a time slot... BTW You can vote for my presentation at: http://dev.extranet.lisasoft.com/foss4g/itemrate/list/all/gdal/ Best regards, Klokan Petr Pridal On Wed, Jun 17, 2009 at 8:00 PM, Trent M Hare<[email protected]> wrote: > > huh - processing using Make - clever. > > We are researching several different approaches for this also but I imagine > we will use Conductor, created by (UofA). Another of our commercial systems > uses Condor (open source - http://www.cs.wisc.edu/condor/ ) . > > Currently the UofA Conductor HiRISE pipeline uses our ISIS image processing > software which is not yet multi-threaded. Thus this system should also work > well for GDAL processing. They also have another tool that monitors > Conductor called Maestro. Don't jump into this lightly as I imagine it will > be tricky to setup (it also requires a database - MySQL). > http://pirlwww.lpl.arizona.edu/software/Conductor.shtml BTW, they are a > completely *nix house (Solaris, Mac OS-X, FreeBSD and Linux). > > here is an abstract (by the author B. Castalia) > http://www.lsstmail.org/meet_coll/ongoing/Software/ > > Regards, > Trent > > > From: Fabrice LALLAURET - PSV <[email protected]> > To: > [email protected] > Date: 06/16/2009 08:32 AM > Subject: Re: [gdal-dev] Distributed processing > Sent by: > [email protected] > ________________________________ > > > John Donovan a écrit : >> Hi, >> We currently have a stand-alone app that converts, mosaicks and scales >> GDAL-supported images to a proprietary format. It works well, but we >> handle tens of thousands of source files at a time, which can slow the >> process down to a crawl. >> >> So we're investigating parallelising this process over several machines, >> and I was wondering if anyone has any experience of this that they'd be >> willing to share? It's still early days yet, so we're open to all >> suggestions. >> >> Regards, >> John Donovan - Programmer, Virtalis Ltd. >> >> ______________________________________________________________________ >> This email has been scanned for Virtalis Ltd by the MessageLabs Email >> Security System. >> ______________________________________________________________________ >> _______________________________________________ >> gdal-dev mailing list >> [email protected] >> http://lists.osgeo.org/mailman/listinfo/gdal-dev > Personally, I create Makefile and use distmake for the distribution purpose. > > Regards > _______________________________________________ > gdal-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > _______________________________________________ > gdal-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/gdal-dev > -- http://blog.klokan.cz/ http://www.maptiler.org/ http://www.oldmapsonline.org/ _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
