>From reading the documentation more closely, I notice that NUM_THREADS is a >separate bit of functionality from -multithread (which only refers to IO), so >any thoughts of load balancing weren't correct.
I suspect GDAL is simply doing exactly as instructed - spawning X threads, even if it won't find work for them. It's therefore up to the user (me!) to measure and work out where the best performance comes from - and blindly throwing in NUM_THREADS=ALL_CPUS is not always a good tactic. Dr Daniel Evans Software Developer From: Daniel Evans Sent: 06 October 2020 14:38 To: [email protected] Subject: GDAL Warp - NUM_THREADS=ALL_CPUS detrimental? Hello, I've found today that when calling gdal.Warp (from Python, though I doubt it matters) with multithreading enabled, setting the option NUM_THREADS=ALL_CPUS results in increased runtime. I find there to be a minimum runtime of ~0.3-0.4s with NUM_THREADS set like this, compared to a runtime of order 0.01-0.03s (measurements aren't particularly accurate) for some tiny test datasets, which is much closer to my expectation. Is this behaviour expected? It may well be that GDAL has some internal logic to optimise the number of threads, and setting NUM_THREADS=ALL_CPUS is detrimentally overriding that. I have often seen code examples or guides that suggest setting NUM_THREADS whenever enabling multithreading in GDAL calls. Is this actually poor advice, with GDAL already using the maximum number of available CPUs when it is suitable to do so? (Runtimes measured in GDAL 3.0.4) Dr Daniel Evans Software Developer [JBA COVID-19 statement]<https://www.jbagroup.co.uk/sites/www.jbagroup.co.uk/files/documents/15-030%20JBA%20Business%20Continuity%20Briefing%20-%20Latest.pdf> T +44 (0) 1756 799919 www.jbarisk.com<http://www.jbarisk.com> [Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png] [Follow us on Twitter] <https://twitter.com/jbarisk> Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT> The JBA Group supports the JBA Trust. All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
