Re: [Qgis-user] [QGIS-Developer] timzonefinder vs point in polygon
Seems there are at least two questions to have a look at. To do a timezone lookup you need a relatively large datafile for the timezones. That could be part of a plugin/tool, or a separate file. Either way, you need to have that data stored and accessible somewhere. And it probably needs updating now and then. Imho plugins should only provide the algorithm and some sort of ui. And maybe, when the plugin is always using the same dataset within it's algorithm, that dataset. Unless it is too large, and that was as I remember the first question you asked the user list Calvin. And that is still pending. I have no problems downloading and storing a dataset, configure the plugin so it knows where I stored the dataset and have the plugin use that dataset whenever I would need t do a timezonze lookup. The second question is finding and using the fastest algorithm. If the Vector=>Datamanagement Tools->Join Attributes by Location is the fastest (is it indeed? cross platform, independent of hardware), is there a way to use that processing tool, which is part of the core tool set , within your datetimetools plugin? I have no experience with building a plugin, but I would hope integrating (or calling on?) tools from the processing toolbox within a plugin should be supported. Seems weird you would have to rewrite an existing tool for it to be used by a plugin. Jeroen -Oorspronkelijk bericht- Van: Nyall Dawson Verzonden: dinsdag 9 februari 2021 23:42 Aan: Matthias Kuhn CC: C Hamilton ; Groene Bij ; qgis-user ; Andrea Giudiceandrea ; qgis-developer Onderwerp: Re: [QGIS-Developer] timzonefinder vs point in polygon On Wed, 10 Feb 2021 at 03:32, Matthias Kuhn wrote: > > Hi Calvin, > > If you are looking for speed I would consider putting things into memory. > I have seen that timezonefinder has an in_memory option which makes it even > faster. > On QGIS side you can also load features into a memory layer and add a spatial > index to that. Just to add to this -- if you pre-prepared the timezone polygon geometries and cached all the results then that's the fastest speed you'll ever achieve :) (at the cost of memory usage, however!) Nyall > > Hope that helps to squeeze the last bit of performance out of this > Matthias > > On Tue, Feb 9, 2021 at 5:52 PM C Hamilton wrote: >> >> It is highly likely that you have a much faster machine than mine so I can >> only look at comparative times between timezonefinder and the PyQGIS code >> included below. You will notice that I take a snapshot of the time before >> and after I iterate through the 10,000 points using both timezonefinder as a >> look up and the code Nyall proposed. That is how I am getting the times. >> What I am really trying to determine is the fastest way to look up a point >> using either timezonefinder or PyQGIS code. Perhaps what I have with PyQGIS >> code is fast enough, but I am not satisfied if it is not close to the speed >> of timezonefinder. >> >> Ultimately, I am trying to determine whether I should continue to use >> timezonefinder as the method to look up time zones in my datetimetools >> plugin or should I have the gpkg time zone polygon layer as a part of the >> plugin and do lookups using it. Right now I am just doing single point >> lookups and it is probably fast enough, but I don't like that it is slower >> than timezonefinder. I am likely to expand datetimetools plugin to have a >> processing algorithm to do a lookup on a whole layer and in that case the >> PyQGIS method is better. >> >> Those are my thoughts and if there is a faster method using the gpkg time >> zone polygon to do the lookup, let me know. One thing I just tried was >> using the shapefile from >> https://github.com/evansiroky/timezone-boundary-builder. It appears it is >> not spatially indexed and was taking too long. I resaved it as a shapefile >> and made sure it had a spatial index, but the gpkg conversion of the >> shapefile is even faster to access. >> >> Calvin >> >> On Tue, Feb 9, 2021 at 11:19 AM wrote: >>> >>> Hi Calvin, >>> >>> >>> >>> Thanks for sending me this from the Developers list. I am not a part of the >>> Developer list, so I did not see the Join attribute by location tip from >>> Andrea. >>> >>> >>> >>> Anyway, what do you mean with not getting fast result with single point >>> lookups? >>> >>> When I select a single point from the 1 point layer and run the same >>> algorithm (check the selected features only option in the dialogue box), >>> the result takes 0.05 seconds (12 seconds for all 1). At least that is >>> what is says in the log tab of the algorithm. And it sure looks quick. >>> >>> >>> >>> Jeroen >>> >>> >>> >>> >>> >>> Van: C Hamilton >>> Verzonden: dinsdag 9 februari 2021 16:25 >>> Aan: Nyall Dawson >>> CC: Andrea Giudiceandrea ; qgis-developer >>> ; Groene Bij >>> Onderwerp: Re: [QGIS-Developer] timzonefinder vs point in polygon >>> >>> >>> >>> Nyall, >>> >>> >>> >>>
Re: [Qgis-user] [QGIS-Developer] timzonefinder vs point in polygon
On Wed, 10 Feb 2021 at 03:32, Matthias Kuhn wrote:
>
> Hi Calvin,
>
> If you are looking for speed I would consider putting things into memory.
> I have seen that timezonefinder has an in_memory option which makes it even
> faster.
> On QGIS side you can also load features into a memory layer and add a spatial
> index to that.
Just to add to this -- if you pre-prepared the timezone polygon
geometries and cached all the results then that's the fastest speed
you'll ever achieve :) (at the cost of memory usage, however!)
Nyall
>
> Hope that helps to squeeze the last bit of performance out of this
> Matthias
>
> On Tue, Feb 9, 2021 at 5:52 PM C Hamilton wrote:
>>
>> It is highly likely that you have a much faster machine than mine so I can
>> only look at comparative times between timezonefinder and the PyQGIS code
>> included below. You will notice that I take a snapshot of the time before
>> and after I iterate through the 10,000 points using both timezonefinder as a
>> look up and the code Nyall proposed. That is how I am getting the times.
>> What I am really trying to determine is the fastest way to look up a point
>> using either timezonefinder or PyQGIS code. Perhaps what I have with PyQGIS
>> code is fast enough, but I am not satisfied if it is not close to the speed
>> of timezonefinder.
>>
>> Ultimately, I am trying to determine whether I should continue to use
>> timezonefinder as the method to look up time zones in my datetimetools
>> plugin or should I have the gpkg time zone polygon layer as a part of the
>> plugin and do lookups using it. Right now I am just doing single point
>> lookups and it is probably fast enough, but I don't like that it is slower
>> than timezonefinder. I am likely to expand datetimetools plugin to have a
>> processing algorithm to do a lookup on a whole layer and in that case the
>> PyQGIS method is better.
>>
>> Those are my thoughts and if there is a faster method using the gpkg time
>> zone polygon to do the lookup, let me know. One thing I just tried was
>> using the shapefile from
>> https://github.com/evansiroky/timezone-boundary-builder. It appears it is
>> not spatially indexed and was taking too long. I resaved it as a shapefile
>> and made sure it had a spatial index, but the gpkg conversion of the
>> shapefile is even faster to access.
>>
>> Calvin
>>
>> On Tue, Feb 9, 2021 at 11:19 AM wrote:
>>>
>>> Hi Calvin,
>>>
>>>
>>>
>>> Thanks for sending me this from the Developers list. I am not a part of the
>>> Developer list, so I did not see the Join attribute by location tip from
>>> Andrea.
>>>
>>>
>>>
>>> Anyway, what do you mean with not getting fast result with single point
>>> lookups?
>>>
>>> When I select a single point from the 1 point layer and run the same
>>> algorithm (check the selected features only option in the dialogue box),
>>> the result takes 0.05 seconds (12 seconds for all 1). At least that is
>>> what is says in the log tab of the algorithm. And it sure looks quick.
>>>
>>>
>>>
>>> Jeroen
>>>
>>>
>>>
>>>
>>>
>>> Van: C Hamilton
>>> Verzonden: dinsdag 9 februari 2021 16:25
>>> Aan: Nyall Dawson
>>> CC: Andrea Giudiceandrea ; qgis-developer
>>> ; Groene Bij
>>> Onderwerp: Re: [QGIS-Developer] timzonefinder vs point in polygon
>>>
>>>
>>>
>>> Nyall,
>>>
>>>
>>>
>>> Using "Join attributes by location" on a large set of points is fast, but
>>> if you are doing single point lookups it bothers me that I am not getting
>>> that fast of results in comparison to timezonefinder. The 10,000 points is
>>> intended to test the speed of many single point lookups. If I were going to
>>> actually do 10,000 point lookups I would use the Join attributes by
>>> location, but here are my results of single point lookups using
>>> timezonefinder vs. your method. Note that the assumption is that they can
>>> be anywhere in the EPSG:4326 bounding box. I am not trying to constrain
>>> them to the QGIS canvas.
>>>
>>>
>>>
>>> timezonefinder: 44 seconds
>>>
>>> Your method: 144 seconds
>>>
>>>
>>>
>>> Here is my code for each:
>>>
>>>
>>>
>>> timezonefinder
>>>
>>> from timezonefinder import TimezoneFinder
>>> import time
>>> tf = TimezoneFinder()
>>>
>>> l = iface.activeLayer()
>>> features = l.getFeatures()
>>>
>>> start_time = time.time()
>>> for f in features:
>>> pt = f.geometry().asPoint()
>>> name = tf.certain_timezone_at(lng=pt.x(), lat=pt.y())
>>> print('Time {}'.format(time.time() - start_time))
>>>
>>>
>>>
>>> Your method:
>>>
>>> import time
>>> tzlayer = QgsVectorLayer("W:\\My
>>> Documents\\GitHub\\timezone_speed_lookup_test\\timezones.gpkg",
>>> "timezones", "ogr")
>>>
>>> l = iface.activeLayer()
>>> features = l.getFeatures()
>>>
>>> start_time = time.time()
>>> for f in features:
>>> pt = f.geometry().asPoint()
>>> rect = QgsRectangle( pt.x()-0.01, pt.y()-0.1,
>>> pt.x(),pt.y())
>>> request = QgsFeatureRequest().setFilterRect(rect)
>>>
Re: [Qgis-user] [QGIS-Developer] timzonefinder vs point in polygon
It is highly likely that you have a much faster machine than mine so I can
only look at comparative times between timezonefinder and the PyQGIS code
included below. You will notice that I take a snapshot of the time before
and after I iterate through the 10,000 points using both timezonefinder as
a look up and the code Nyall proposed. That is how I am getting the times.
What I am really trying to determine is the fastest way to look up a point
using either timezonefinder or PyQGIS code. Perhaps what I have with PyQGIS
code is fast enough, but I am not satisfied if it is not close to the speed
of timezonefinder.
Ultimately, I am trying to determine whether I should continue to use
timezonefinder as the method to look up time zones in my
datetimetools plugin or should I have the gpkg time zone polygon layer as a
part of the plugin and do lookups using it. Right now I am just doing
single point lookups and it is probably fast enough, but I don't like that
it is slower than timezonefinder. I am likely to expand datetimetools
plugin to have a processing algorithm to do a lookup on a whole layer and
in that case the PyQGIS method is better.
Those are my thoughts and if there is a faster method using the gpkg time
zone polygon to do the lookup, let me know. One thing I just tried was
using the shapefile from
https://github.com/evansiroky/timezone-boundary-builder. It appears it is
not spatially indexed and was taking too long. I resaved it as a shapefile
and made sure it had a spatial index, but the gpkg conversion of the
shapefile is even faster to access.
Calvin
On Tue, Feb 9, 2021 at 11:19 AM wrote:
> Hi Calvin,
>
>
>
> Thanks for sending me this from the Developers list. I am not a part of
> the Developer list, so I did not see the Join attribute by location tip
> from Andrea.
>
>
>
> Anyway, what do you mean with not getting fast result with single point
> lookups?
>
> When I select a single point from the 1 point layer and run the same
> algorithm (check the selected features only option in the dialogue box),
> the result takes 0.05 seconds (12 seconds for all 1). At least that is
> what is says in the log tab of the algorithm. And it sure looks quick.
>
>
>
> Jeroen
>
>
>
>
>
> *Van:* C Hamilton
> *Verzonden:* dinsdag 9 februari 2021 16:25
> *Aan:* Nyall Dawson
> *CC:* Andrea Giudiceandrea ; qgis-developer <
> qgis-develo...@lists.osgeo.org>; Groene Bij
> *Onderwerp:* Re: [QGIS-Developer] timzonefinder vs point in polygon
>
>
>
> Nyall,
>
>
>
> Using "Join attributes by location" on a large set of points is fast, but
> if you are doing single point lookups it bothers me that I am not getting
> that fast of results in comparison to timezonefinder. The 10,000 points is
> intended to test the speed of many single point lookups. If I were going to
> actually do 10,000 point lookups I would use the Join attributes by
> location, but here are my results of single point lookups using
> timezonefinder vs. your method. Note that the assumption is that they can
> be anywhere in the EPSG:4326 bounding box. I am not trying to constrain
> them to the QGIS canvas.
>
>
>
> timezonefinder: 44 seconds
>
> Your method: 144 seconds
>
>
>
> Here is my code for each:
>
>
>
> *timezonefinder*
>
> from timezonefinder import TimezoneFinder
> import time
> tf = TimezoneFinder()
>
> l = iface.activeLayer()
> features = l.getFeatures()
>
> start_time = time.time()
> for f in features:
> pt = f.geometry().asPoint()
> name = tf.certain_timezone_at(lng=pt.x(), lat=pt.y())
> print('Time {}'.format(time.time() - start_time))
>
>
>
> *Your method*:
>
> import time
> tzlayer = QgsVectorLayer("W:\\My
> Documents\\GitHub\\timezone_speed_lookup_test\\timezones.gpkg",
> "timezones", "ogr")
>
> l = iface.activeLayer()
> features = l.getFeatures()
>
> start_time = time.time()
> for f in features:
> pt = f.geometry().asPoint()
> rect = QgsRectangle( pt.x()-0.01, pt.y()-0.1,
> pt.x(),pt.y())
> request = QgsFeatureRequest().setFilterRect(rect)
> zones_intersecting_bounding_box = tzlayer.getFeatures(request)
> for tz_feature in zones_intersecting_bounding_box:
> if tz_feature.geometry().intersects(rect):
> name = tz_feature[1]
> break
>
> print('Time {}'.format(time.time() - start_time))
>
>
>
>
>
> I'd suggest using similar logic to what join attributes by location
> does internally. In pseudocode:
>
> search_point = ...
> search_bbox = QgsRectangle( search_point.x() - some tolerance,
> search_point.y() - some_tolerance, ... )
> request = QgsFeatureRequest().setFilterRect(search_bbox)
> zones_intersecting_bounding_box = tz_layer.getFeatures(request)
> for time_zone_feature in zones_intersecting_bounding_box:
> if time_zone_feature.intersects(search_point):
> # found a hit!
> break
>
> This will take advantage of whatever spatial index you have on the
> time zone layer (e.g. the internal shapefile/gpkg/... index)
>
> There's quite a
Re: [Qgis-user] [QGIS-Developer] timzonefinder vs point in polygon
Hi Calvin,
Thanks for sending me this from the Developers list. I am not a part of the
Developer list, so I did not see the Join attribute by location tip from Andrea.
Anyway, what do you mean with not getting fast result with single point lookups?
When I select a single point from the 1 point layer and run the same
algorithm (check the selected features only option in the dialogue box), the
result takes 0.05 seconds (12 seconds for all 1). At least that is what is
says in the log tab of the algorithm. And it sure looks quick.
Jeroen
Van: C Hamilton
Verzonden: dinsdag 9 februari 2021 16:25
Aan: Nyall Dawson
CC: Andrea Giudiceandrea ; qgis-developer
; Groene Bij
Onderwerp: Re: [QGIS-Developer] timzonefinder vs point in polygon
Nyall,
Using "Join attributes by location" on a large set of points is fast, but if
you are doing single point lookups it bothers me that I am not getting that
fast of results in comparison to timezonefinder. The 10,000 points is intended
to test the speed of many single point lookups. If I were going to actually do
10,000 point lookups I would use the Join attributes by location, but here are
my results of single point lookups using timezonefinder vs. your method. Note
that the assumption is that they can be anywhere in the EPSG:4326 bounding box.
I am not trying to constrain them to the QGIS canvas.
timezonefinder: 44 seconds
Your method: 144 seconds
Here is my code for each:
timezonefinder
from timezonefinder import TimezoneFinder
import time
tf = TimezoneFinder()
l = iface.activeLayer()
features = l.getFeatures()
start_time = time.time()
for f in features:
pt = f.geometry().asPoint()
name = tf.certain_timezone_at(lng=pt.x(), lat=pt.y())
print('Time {}'.format(time.time() - start_time))
Your method:
import time
tzlayer = QgsVectorLayer("W:\\My
Documents\\GitHub\\timezone_speed_lookup_test\\timezones.gpkg", "timezones",
"ogr")
l = iface.activeLayer()
features = l.getFeatures()
start_time = time.time()
for f in features:
pt = f.geometry().asPoint()
rect = QgsRectangle( pt.x()-0.01, pt.y()-0.1, pt.x(),pt.y())
request = QgsFeatureRequest().setFilterRect(rect)
zones_intersecting_bounding_box = tzlayer.getFeatures(request)
for tz_feature in zones_intersecting_bounding_box:
if tz_feature.geometry().intersects(rect):
name = tz_feature[1]
break
print('Time {}'.format(time.time() - start_time))
I'd suggest using similar logic to what join attributes by location
does internally. In pseudocode:
search_point = ...
search_bbox = QgsRectangle( search_point.x() - some tolerance,
search_point.y() - some_tolerance, ... )
request = QgsFeatureRequest().setFilterRect(search_bbox)
zones_intersecting_bounding_box = tz_layer.getFeatures(request)
for time_zone_feature in zones_intersecting_bounding_box:
if time_zone_feature.intersects(search_point):
# found a hit!
break
This will take advantage of whatever spatial index you have on the
time zone layer (e.g. the internal shapefile/gpkg/... index)
There's quite a number of extra optimisations which could be done here
if the speed isn't sufficient for mouse movements, e.g. you could
fetch all the zone features in the canvas extent and "prepare" their
geometries for ultra fast point in polygon tests, and then invalidate
this cache and rebuild on each canvas extent change. But I'd only do
that kind of thing if needed...
Nyall
>
> Calvin
>
> On Fri, Feb 5, 2021 at 6:30 PM Nyall Dawson > wrote:
>>
>> On Sat, 6 Feb 2021 at 07:54, Andrea Giudiceandrea > > wrote:
>> >
>> > C Hamilton wrote
>> > > Using the algorithm Vector->Geoprocessing Tools->Intersection: 4 minutes,
>> > > 4
>> > > seconds
>> >
>> > Hi Calvin,
>> > maybe it could be better to use the "Join attributes by location"
>> > algorithm.
>> > It only takes few seconds to preform the join.
>>
>> Confirmed - for me it only takes ~2 seconds, and that's on an
>> non-optimised debug build! There may be a few % more performance boost
>> on the proper QGIS release builds.
>>
>> "Join attributes by location" has a bunch of extra logic to optimise
>> the method that the join is performed, which really pays off in this
>> particular situation (matching points to polygons, where number of
>> points >> number of polygons).
>>
>> Nyall
>>
>>
>>
>> >
>> > Regards.
>> >
>> > Andrea
>> >
>> >
>> >
>> > --
>> > Sent from: http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-f4099106.html
>> > ___
>> > QGIS-Developer mailing list
>> > qgis-develo...@lists.osgeo.org
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> ___
>> QGIS-Developer mailing list
