Hi Sean, That seems to be the best option. I was testing on Shapefile initially, but after I posted, I tried a memory layer and got quite different results. Ultimately, our output format is determined by clients (it's usually Shapefile or File GDB), so we'll just have to let them know there are performance implications with some formats.
Thanks, Jon e: [email protected] t: +44 (0)1756 799919 www.jbarisk.com All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, England. From: gdal-dev <[email protected]> On Behalf Of Sean Gillies Sent: 20 January 2022 15:44 To: [email protected] Subject: Re: [gdal-dev] Faster alternative to GetFeatureCount? CAUTION: This email originated from outside of JBA and contains one or more links and one or more attachments. DO NOT click links or open attachments unless you recognise the sender's email address and are absolutely certain that the content is safe. See the Phishing page on IMS on SP for more information about how to spot and report suspicious messages. WARNING: This email contains one or more attachments with links within the attachment(s). Please take extra care when handling the attachments. Hi Jon, The performance of GetFeatureCount, filters, and GetNextFeature depends a lot on the data format. If your data is in Postgres, for example, these operations are super fast because the database keeps the number of rows (features) in memory and has highly optimized views of selected rows. If your data is in a CSV file, GDAL has to read every line to count the number of features and when getting filtered features has to read all the lines in between the ones you want. See https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/csv/ogrcsvlayer.cpp#L1667. Can you switch to a more optimized format? On Thu, Jan 20, 2022 at 2:02 AM Jon Morris <[email protected]<mailto:[email protected]>> wrote: I'm writing applications using the GDAL Python bindings and when I profile for performance, GetFeatureCount frequently comes out near the top. I'm often using it to check whether a spatial or attribute filter has returned any features and don't need the full count. When the layer contains millions of features, there would be a big performance improvement if we could exit the count early. Is there a better way of doing this? I've tried using GetNextFeature instead, but there must be quite a lot of overhead in that function as it is much slower. All I need to know is if the layer has has 0, 1 or >1 features, I don't need the actual count. Can anyone suggest the fastest way of doing this in Python? I'm using GDAL 3.3.1 at the moment but could upgrade if there is new functionality that would help. Thanks, Jon Jon Morris Software Developer e: [email protected]<mailto:[email protected]> t: +44 (0)1756 799919<tel:+44%20(0)1756%20799919> www.jbarisk.com<http://www.jbarisk.com/> [cid:[email protected]] [Facebook]<https://www.facebook.com/TheFloodPeople> [LinkedIn]<https://www.linkedin.com/company/jba-risk-management/> [Twitter]<https://twitter.com/JBARisk> [YouTube]<https://www.youtube.com/channel/UC0iatom2jYbW96voW0rlpCw> All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, England. -- Sean Gillies
_______________________________________________ gdal-dev mailing list [email protected] https://lists.osgeo.org/mailman/listinfo/gdal-dev
