Hi, We currently do parallel execution of ArGIS geoprocessing tools, although not yet with GDAL. We use IPython to execute normal commands in parallel. IPython is a shell replacement that has parallel commands that can be used from the shell or within code.
The biggest difficulty I have found is using resources efficiently. You will inevitably hit a bottleneck of either cpu speed, disk bandwidth, network bandwidth or memory. It is worth spending some time thinking about what resources a particular process will consume, and where bottlenecks are likely to occur. -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of John Donovan Sent: 12 June 2009 12:06 To: [email protected] Subject: [gdal-dev] Distributed processing Hi, We currently have a stand-alone app that converts, mosaicks and scales GDAL-supported images to a proprietary format. It works well, but we handle tens of thousands of source files at a time, which can slow the process down to a crawl. So we're investigating parallelising this process over several machines, and I was wondering if anyone has any experience of this that they'd be willing to share? It's still early days yet, so we're open to all suggestions. Regards, John Donovan - Programmer, Virtalis Ltd. ______________________________________________________________________ This email has been scanned for Virtalis Ltd by the MessageLabs Email Security System. ______________________________________________________________________ _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev This e-mail message and any attached file is the property of the sender and is sent in confidence to the addressee only. The contents are not to be disclosed to anyone other than the addressee. Unauthorised recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission. If you experience difficulty with opening any attachments to this message, or with sending a reply by email, please telephone on + 44-(0)1235 438151 or fax on + 44-(0)1235 438188. Any advice contained in this e-mail or any accompanying file attached hereto is for information purposes only. RPS do not take any responsibility for differences between the original and the transmission copy or any amendments made thereafter. If the addressee requires RPS to be responsible for the contents of this e-mail, RPS will be pleased to issue a signed hard copy of the document upon request. RPS Planning and Development Limited, company number: 02947164 (England). Registered office: Centurion Court, 85 Milton Park Abingdon Oxfordshire OX14 4RY. RPS Group Plc web link: <http://www.rpsgroup.com> _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
