Re: [Qgis-user] Optimize QGIS Command_Line
Originally its in MSSQL, we have done a makevalid() on all lines before we use OGR2OGR to export it to a GeoPackage. That file is on the local NVME drive. We did try setting the spatial index to no, this had very little difference in time. Dennis Burgess <https://mail.linktechs.net/owa/redir.aspx?C=AWq0kPmLomFxbshE65wWiGsm9JYjRq7NfAx66HZMGnN2I3lAPi_UCA..=mailto%3admburgess%40linktechs.net> From: Frank Broniewski Sent: Wednesday, February 7, 2024 12:35:52 AM To: QGIS-User@lists.osgeo.org; Dennis Burgess Subject: AW: [Qgis-user] Optimize QGIS Command_Line Hi Dennis, concerning your first command, is your GeoPackage stored locally or on a network drive? Writing SQLite/GepPackage databases onto a network drive can have a severe performance impact. Another idea, did you try setting -lco SPATIAL_INDEX=no and creating the index after the import? This might speed up your import as well. This page https://gdal.org/drivers/vector/sqlite.html has some more insights on performance, but maybe you know it already ... Concerning your second command, without knowing how many lines / polygons you want to dissolve here (seems like a lot looking at the file size) it looks pretty normal to me. Keep in mind that dissolving is a rather cost intensive process, when not dissolving by attribute. As every geometry has to be checked against all the others if they can be dissolved spatially - spatial indexes aside. I do not know the internals of the algorithm itself, but I would guess it is not well suited for a threaded / multi process approach and it just runs on one core of your CPU doing all the work - like most GIS tools do. A speedier approach could be to spatially divide your data into smaller parts, do the dissolve on each part to reduce the number of features, and then finally dissolve the results into the final one ... Von: QGIS-User im Auftrag von Dennis Burgess via QGIS-User Gesendet: Dienstag, 6. Februar 2024 15:23 An: QGIS-User@lists.osgeo.org Betreff: Re: [Qgis-user] Optimize QGIS Command_Line ogr2ogr -f gpkg geom_multi_38.gpkg "MSSQL:server=xx;database=x;uid=x;pwd=;MultipleActiveResultSets=true;" -sql "EXEC xx 42039, 38" -overwrite -gt 100 -lco SPATIAL_INDEX=yes OGR_SQLITE_CACHE=10240MB OGR_SQLITE_SYNCHRONOUS=OFF This one takes around 30 min to export to gpkg file, if I run the same exec command on sql it takes less than 40 seconds to pull down .. these are geometry lines. Around 940 lines. The second one "c:\program files\qgis 3.26.2\apps\qgis\bin\qgis_process" run native:dissolve --INPUT=geom_multi_55096.gpkg --OUTPUT=geom_multi_diss_38.gpkg This would e one of them. Last job took 49 minutes, orginal input is around 1.2 gig. Dedicated platform for processing: 72 cores, 32 gig of RAM, running on NVME disk. SQL 2019 Enterprise: 80 cores, 512gig ram, NVME disks. From: Johannes Kröger (WhereGroup) Sent: Tuesday, February 6, 2024 2:01 AM To: Dennis Burgess ; QGIS-User@lists.osgeo.org Subject: Re: [Qgis-user] Optimize QGIS Command_Line Hey Dennis, can you share those command lines (maybe with sensitive information stripped)? Otherwise it is hard to guess what the issue might be. What does your CPU, RAM and I/O utilisation look like during those long running processes? Same for the MSSQL database server host system. Cheers, Hannes On 05.02.24 14:46, Dennis Burgess via QGIS-User wrote: I have two command lines, one reads from my MSSQL database and creates a GeoPackage on the local disk. This takes WAY too long, just to create it , it uses 5% or less CPU, and is SUPER slow. Looking for methods to speed this up. The data is around 1.2 to 1.4 gig of pop, but it should not take 50 min to create. Also I am doing a dissolve on this data, once its on the disk, this also takes quite a while, even though when I do it from QGIS it takes like 10 min vs an hour or more on the computer with the command line. I’m sure I am doing something wrong on both of these. Just trying to understand what it would be. Thanks for any assistance ! Dennis ___ QGIS-User mailing list QGIS-User@lists.osgeo.org<mailto:QGIS-User@lists.osgeo.org> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user -- Johannes Kröger / GIS-Entwickler/-Berater - Aufwind durch Wissen! Web-Seminare und Online-Schulungen bei der https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=www.foss%2dacademy.com=675F481A-110D-D506-B43D-B6ED42E994FC=079c058f437b7c6303d36c6513e5e8848d0c5ac4-2fd8998bef3b8d7c17576198b1589c011d7f32ce<https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=http%3a%2f%2fwww.foss%2daca
Re: [Qgis-user] Optimize QGIS Command_Line
ogr2ogr -f gpkg geom_multi_38.gpkg "MSSQL:server=xx;database=x;uid=x;pwd=;MultipleActiveResultSets=true;" -sql "EXEC xx 42039, 38" -overwrite -gt 100 -lco SPATIAL_INDEX=yes OGR_SQLITE_CACHE=10240MB OGR_SQLITE_SYNCHRONOUS=OFF This one takes around 30 min to export to gpkg file, if I run the same exec command on sql it takes less than 40 seconds to pull down .. these are geometry lines. Around 940 lines. The second one "c:\program files\qgis 3.26.2\apps\qgis\bin\qgis_process" run native:dissolve --INPUT=geom_multi_55096.gpkg --OUTPUT=geom_multi_diss_38.gpkg This would e one of them. Last job took 49 minutes, orginal input is around 1.2 gig. Dedicated platform for processing: 72 cores, 32 gig of RAM, running on NVME disk. SQL 2019 Enterprise: 80 cores, 512gig ram, NVME disks. From: Johannes Kröger (WhereGroup) Sent: Tuesday, February 6, 2024 2:01 AM To: Dennis Burgess ; QGIS-User@lists.osgeo.org Subject: Re: [Qgis-user] Optimize QGIS Command_Line Hey Dennis, can you share those command lines (maybe with sensitive information stripped)? Otherwise it is hard to guess what the issue might be. What does your CPU, RAM and I/O utilisation look like during those long running processes? Same for the MSSQL database server host system. Cheers, Hannes On 05.02.24 14:46, Dennis Burgess via QGIS-User wrote: I have two command lines, one reads from my MSSQL database and creates a GeoPackage on the local disk. This takes WAY too long, just to create it , it uses 5% or less CPU, and is SUPER slow. Looking for methods to speed this up. The data is around 1.2 to 1.4 gig of pop, but it should not take 50 min to create. Also I am doing a dissolve on this data, once its on the disk, this also takes quite a while, even though when I do it from QGIS it takes like 10 min vs an hour or more on the computer with the command line. I’m sure I am doing something wrong on both of these. Just trying to understand what it would be. Thanks for any assistance ! Dennis ___ QGIS-User mailing list QGIS-User@lists.osgeo.org<mailto:QGIS-User@lists.osgeo.org> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user -- Johannes Kröger / GIS-Entwickler/-Berater - Aufwind durch Wissen! Web-Seminare und Online-Schulungen bei der https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=www.foss%2dacademy.com=7036B80A-10B7-5306-BAB5-505120F413E8=079c058f437b7c6303d36c6513e5e8848d0c5ac4-1d48a351cf96dc78fcc69089ad0b4a4cef7388e2<https://imsva91-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=http%3a%2f%2fwww.foss%2dacademy.com=7036B80A-10B7-5306-BAB5-505120F413E8=079c058f437b7c6303d36c6513e5e8848d0c5ac4-0b980f826f3d937e36ac4d15a89c87a459c73be8> - WhereGroup GmbH c/o KK03 GmbH Lange Reihe 29 20099 Hamburg Germany Tel: +49 (0)228 / 90 90 38 - 36 Fax: +49 (0)228 / 90 90 38 - 11 johannes.kroe...@wheregroup.com<mailto:johannes.kroe...@wheregroup.com> www.wheregroup.com<http://www.wheregroup.com> Geschäftsführer: Olaf Knopp, Peter Stamm Amtsgericht Bonn, HRB 9885 --- ___ QGIS-User mailing list QGIS-User@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] Optimize QGIS Command_Line
I have two command lines, one reads from my MSSQL database and creates a GeoPackage on the local disk. This takes WAY too long, just to create it , it uses 5% or less CPU, and is SUPER slow. Looking for methods to speed this up. The data is around 1.2 to 1.4 gig of pop, but it should not take 50 min to create. Also I am doing a dissolve on this data, once its on the disk, this also takes quite a while, even though when I do it from QGIS it takes like 10 min vs an hour or more on the computer with the command line. I'm sure I am doing something wrong on both of these. Just trying to understand what it would be. Thanks for any assistance ! Dennis ___ QGIS-User mailing list QGIS-User@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] GPKG Multi-Layer to one flat
Many overlaps, just want if the area is covered, to show. I really just used a Disssolve, this reducded the number of polygons to less than ½ and is fairly quick.. Quciker than Union. [LTI-Full_175px] Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage’s with www.towercoverage.com From: chris hermansen Sent: Friday, March 3, 2023 1:58 PM To: Dennis Burgess Cc: qgis-user@lists.osgeo.org Subject: Re: [Qgis-user] GPKG Multi-Layer to one flat Dennis and list On Fri, Mar 3, 2023 at 11:45 AM Dennis Burgess via QGIS-User mailto:qgis-user@lists.osgeo.org>> wrote: Ok, has to be a simple solution. I have a GPKG file that is around 50 meg. This has multiple layers or at least I can see multiple layers. I.e. I have one polygon on top of another What I want is to flatten these, to where ONLY the exact area that is show is displayed in the smallest file possible. Union is the way to do this but its SLOW SLOW.. any other options… ? Are the overlapping polygon geometries "the same", ie boundaries are coincident? If that's the case, then the attributes from each polygon refer to exactly the same and you could just pick one of the layers and join the attributes on from the other layers. If the geometries are distinct, union is one choice. Another might be to convert all the polygons to linestrings, merge the linestrings into one layer, convert the merged linestrings back to polygons and then use spatial join to transfer the attributes back to the new polygons. Not sure if that would be faster. One potential problem with this general sort of problem is the creation of many tiny polygons where the linework is not quite coincident. This can certainly mess with your concept of "the exact area". Rasterizing the polygons, overlaying, then vectorizing might be another choice. I think it's probably hard to say "what's best" or "what's simplest" without actually seeing an example of the overlap. -- Chris Hermansen · clhermansen "at" gmail "dot" com C'est ma façon de parler. ___ QGIS-User mailing list QGIS-User@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] GPKG Multi-Layer to one flat
Ok, has to be a simple solution. I have a GPKG file that is around 50 meg. This has multiple layers or at least I can see multiple layers. I.e. I have one polygon on top of another What I want is to flatten these, to where ONLY the exact area that is show is displayed in the smallest file possible. Union is the way to do this but its SLOW SLOW.. any other options... ? [LTI-Full_175px] Dennis Burgess Mikrotik : Trainer, Network Associate, Routing Engineer, Wireless Engineer, Traffic Control Engineer, Inter-Networking Engineer, Security Engineer, Enterprise Wireless Engineer Hurricane Electric: IPv6 Sage Level Cambium: ePMP Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Create Wireless Coverage's with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net How did we do today? [Gold Star]<https://app.customerthermometer.com/?template=log_feedback=5badbac1_data=dGVtcGVyYXR1cmVfaWQ9MSZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Green Light]<https://app.customerthermometer.com/?template=log_feedback=675abe04_data=dGVtcGVyYXR1cmVfaWQ9MiZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Yellow Light]<https://app.customerthermometer.com/?template=log_feedback=e42b48a5_data=dGVtcGVyYXR1cmVfaWQ9MyZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Red Light]<https://app.customerthermometer.com/?template=log_feedback=ecaadcd3_data=dGVtcGVyYXR1cmVfaWQ9NCZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===> ___ QGIS-User mailing list QGIS-User@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] Looking for application/server etc
I don't know what it is called. We currently use google maps to display, but we are loading everything onto maps, so it works as long as its not big. We need something that can auto scale depending on the map view.. Dennis Burgess Mikrotik : Trainer, Network Associate, Routing Engineer, Wireless Engineer, Traffic Control Engineer, Inter-Networking Engineer, Security Engineer, Enterprise Wireless Engineer Hurricane Electric: IPv6 Sage Level Cambium: ePMP Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net Create Wireless Coverage's with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net How did we do today? -Original Message- From: johannes.kroe...@wheregroup.com Sent: Thursday, January 12, 2023 10:14 AM To: Dennis Burgess Cc: qgis-user@lists.osgeo.org Subject: Re: [Qgis-user] Looking for application/server etc Vector tiles? Am 2023-01-12 16:43, schrieb Dennis Burgess via QGIS-User: > Looking for something that can take a HUGE, say a 50-75 meg > mutli-polygon (multi-state etc.), that has detail level down to > 1meter, then parse it and display it within 1meg on top of mapping > systems in a web browser ? Options? Not to mention, we have a few > thousand of these so we need to be able to call the mapping for any > one of them. We also need it to simplify it i.e. maybe deliver a max > a 1 meg to the customer.. if they zoom out we want to make it simpler > to deliver something to the customer.. I'm sure there is a way of > doing this.. > > I'm sure there is an application, server, something that can do this > automatically.. > > Dennis Burgess > ___ > QGIS-User mailing list > QGIS-User@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ QGIS-User mailing list QGIS-User@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] Looking for application/server etc
But looking to only display what is on the customer screen. We should never get the 50 meg poly.. [LTI-Full_175px] Dennis Burgess Mikrotik : Trainer, Network Associate, Routing Engineer, Wireless Engineer, Traffic Control Engineer, Inter-Networking Engineer, Security Engineer, Enterprise Wireless Engineer Hurricane Electric: IPv6 Sage Level Cambium: ePMP Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Create Wireless Coverage's with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net How did we do today? [Gold Star]<https://app.customerthermometer.com/?template=log_feedback=5badbac1_data=dGVtcGVyYXR1cmVfaWQ9MSZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Green Light]<https://app.customerthermometer.com/?template=log_feedback=675abe04_data=dGVtcGVyYXR1cmVfaWQ9MiZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Yellow Light]<https://app.customerthermometer.com/?template=log_feedback=e42b48a5_data=dGVtcGVyYXR1cmVfaWQ9MyZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Red Light]<https://app.customerthermometer.com/?template=log_feedback=ecaadcd3_data=dGVtcGVyYXR1cmVfaWQ9NCZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===> From: Richard McDonnell Sent: Thursday, January 12, 2023 10:18 AM To: Dennis Burgess Cc: qgis-user@lists.osgeo.org Subject: RE: Looking for application/server etc PostGIS with Geometry Index's?! Set up Styles with Zoom Levels so as when zoomed in to 1:5000, that's the only time you have full LOD -- Richard McDonnell MSc GIS, FME Certified Professional FRM Data Management -- Oifig na nOibreacha Poiblí Office of Public Works Sráid Jonathan Swift, Baile Átha Troim, Co na Mí, C15 NX36 Jonathan Swift Street, Trim, Co Meath, C15 NX36 -- M +353 87 688 5964 T +353 46 942 2409 https://gov.ie/opw -- To send me files larger than 30MB, please use the link below https://filetransfer.opw.ie/filedrop/richard.mcdonn...@opw.ie Email Disclaimer: https://www.gov.ie/en/organisation-information/439daf-email-disclaimer/ -- MSc GIS, FME Certified Professional -- Oifig na nOibreacha Poiblí Office of Public Works Sráid Jonathan Swift, Baile Átha Troim, Co na Mí, C15 NX36 Jonathan Swift Street, Trim, Co Meath, C15 NX36 -- M +353 87 688 5964 T +353 46 942 2409 https://https://gov.ie/opw<https://www.opw.ie> -- Email Disclaimer: https://www.gov.ie/en/organisation-information/439daf-email-disclaimer/<https://www.opw.ie/en/disclaimer/> From: QGIS-User mailto:qgis-user-boun...@lists.osgeo.org>> On Behalf Of Dennis Burgess via QGIS-User Sent: 12 January 2023 15:43 To: qgis-user@lists.osgeo.org<mailto:qgis-user@lists.osgeo.org> Subject: [Qgis-user] Looking for application/server etc Looking for something that can take a HUGE, say a 50-75 meg mutli-polygon (multi-state etc.), that has detail level down to 1meter, then parse it and display it within 1meg on top of mapping systems in a web browser ? Options? Not to mention, we have a few thousand of these so we need to be able to call the mapping for any one of them. We also need it to simplify it i.e. maybe deliver a max a 1 meg to the customer.. if they zoom out we want to make it simpler to deliver something to the customer.. I'm sure there is a way of doing this.. I'm sure there is an application, server, something that can do this automatically.. Dennis Burgess ___ QGIS-User mailing list QGIS-User@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] Looking for application/server etc
Looking for something that can take a HUGE, say a 50-75 meg mutli-polygon (multi-state etc.), that has detail level down to 1meter, then parse it and display it within 1meg on top of mapping systems in a web browser ? Options? Not to mention, we have a few thousand of these so we need to be able to call the mapping for any one of them. We also need it to simplify it i.e. maybe deliver a max a 1 meg to the customer.. if they zoom out we want to make it simpler to deliver something to the customer.. I'm sure there is a way of doing this.. I'm sure there is an application, server, something that can do this automatically.. Dennis Burgess ___ QGIS-User mailing list QGIS-User@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] QGIS_process not mutli-threaded? or option?
[cid:image002.png@01D8E55B.81B55940] 44 cores with hyperthreading, looks like 1 is used ☹ [LTI-Full_175px] Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage’s with www.towercoverage.com From: Qgis-user On Behalf Of Dennis Burgess via Qgis-user Sent: Friday, October 21, 2022 2:42 PM To: qgis-user@lists.osgeo.org Subject: [Qgis-user] QGIS_process not mutli-threaded? or option? Looks like QGIS_process is not multi-threaded ☹ At least what I am seeing in one core taken up? Any method to speed this thing up? [LTI-Full_175px] Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage’s with www.towercoverage.com ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] QGIS_process not mutli-threaded? or option?
Looks like QGIS_process is not multi-threaded ☹ At least what I am seeing in one core taken up? Any method to speed this thing up? [LTI-Full_175px] Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage’s with www.towercoverage.com ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] ogr2ogr slow
Doing a MSSQL extract, takes my machine 36 seconds to get the data remotely, about 40 meg worth .. but when OGR2OGR gets that data, then does whatever it needs to on GPKG file, it only uses about 5% of the CPU, anyway to use more and complete quicker? [LTI-Full_175px] Dennis Burgess Mikrotik : Trainer, Network Associate, Routing Engineer, Wireless Engineer, Traffic Control Engineer, Inter-Networking Engineer, Security Engineer, Enterprise Wireless Engineer Hurricane Electric: IPv6 Sage Level Cambium: ePMP Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Create Wireless Coverage's with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net How did we do today? [Gold Star]<https://app.customerthermometer.com/?template=log_feedback=5badbac1_data=dGVtcGVyYXR1cmVfaWQ9MSZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Green Light]<https://app.customerthermometer.com/?template=log_feedback=675abe04_data=dGVtcGVyYXR1cmVfaWQ9MiZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Yellow Light]<https://app.customerthermometer.com/?template=log_feedback=e42b48a5_data=dGVtcGVyYXR1cmVfaWQ9MyZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Red Light]<https://app.customerthermometer.com/?template=log_feedback=ecaadcd3_data=dGVtcGVyYXR1cmVfaWQ9NCZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===> ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] Intersect vs extractbylocation
When I tried both methods, the interests takes about 20x times longer than the extract. The number of rows output are the same. So my assumption is that they would be the exact same just one faster. Was just making sure that I did not miss something in it.. [LTI-Full_175px] Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage’s with www.towercoverage.com From: Alexandre Neto Sent: Wednesday, September 14, 2022 10:47 AM To: Dennis Burgess Cc: QGIS User Subject: Re: [Qgis-user] Intersect vs extractbylocation Hi Dennis, The intersects will try to combine the geometries of the two input layers. In your case, it will always return a point anyway. Also, it will add both input layer's attributes in the output file. The Extract by location will just check which points are within the polygons and present it as an output, hence much faster. Alexandre Neto Dennis Burgess via Qgis-user mailto:qgis-user@lists.osgeo.org>> escreveu no dia quarta, 14/09/2022 à(s) 16:20: I have a shape file and a list of points.. Trying to understand what the difference is between intersect vs exteactbylocation . Extract by location takes NO TIME, under a min on all of my runs, but intersection takes quite a bit of time.. In the end, all I want is what lines in the CSV (packaged as GPKG) are within or are contained within the Shapefile. [LTI-Full_175px] Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage’s with www.towercoverage.com<http://www.towercoverage.com> ___ Qgis-user mailing list Qgis-user@lists.osgeo.org<mailto:Qgis-user@lists.osgeo.org> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] Intersect vs extractbylocation
I have a shape file and a list of points.. Trying to understand what the difference is between intersect vs exteactbylocation . Extract by location takes NO TIME, under a min on all of my runs, but intersection takes quite a bit of time.. In the end, all I want is what lines in the CSV (packaged as GPKG) are within or are contained within the Shapefile. [LTI-Full_175px] Dennis Burgess, Mikrotik Certified Trainer MTCNA, MTCRE, MTCWE, MTCTCE, MTCINE, MTCSE, HE IPv6 Sage, Cambium ePMP Certified Author of "Learn RouterOS- Second Edition" Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Need to Automate MikroTik Backups: https://cloud.linktechs.net Create Wireless Coverage's with www.towercoverage.com ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] Faster Intersects
That did it!!! I was unware that GeoPackage’s are SUPER fast and that I could SAVE a CSV as a GeoPackage file. . There is no metho to create indexes though once I do this, but now it runs in about 2 min! lol [LTI-Full_175px] Dennis Burgess Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Create Wireless Coverage’s with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net From: Alexandre Neto Sent: Friday, September 9, 2022 12:20 PM To: Dennis Burgess Cc: Raymond Nijssen ; QGIS User Subject: Re: [Qgis-user] Faster Intersects Following raymond advice, Move the CSV to a shapefile or geopackage and run the create spatial index on it. If you have very complex polygons, use suvdivide to create smaller ones save it in shapefile or geopackage and run create spatial index on it. https://docs.qgis.org/3.22/en/docs/user_manual/processing_algs/qgis/vectorgeometry.html#subdivide I would try with a smaller subset of points to test the improvement speed before trying to run on the full dataset again. Answering your question, yes I am pretty sure postgis is much faster than mssql, and yes I think most recent versions of PostgreSQL PostGIS make use of more than one core. Good luck A sexta, 9/09/2022, 17:59, Dennis Burgess via Qgis-user mailto:qgis-user@lists.osgeo.org>> escreveu: In the format, are you talking about the shape or the CSV with 2million lines? Dennis Burgess Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net Create Wireless Coverage’s with www.towercoverage.com<http://www.towercoverage.com> Need MikroTik Cloud Management: https://cloud.linktechs.net -Original Message- From: Qgis-user mailto:qgis-user-boun...@lists.osgeo.org>> On Behalf Of Raymond Nijssen via Qgis-user Sent: Friday, September 9, 2022 11:11 AM To: qgis-user@lists.osgeo.org<mailto:qgis-user@lists.osgeo.org> Subject: Re: [Qgis-user] Faster Intersects Some things to check: * Is your data local? * Is it in a fast file format? (for example .gpkg or .shp and not .csv or .geojson) * Does the data have a spatial index? * Is the geometry very complicated, like a huge multi polygon with thousands of vertices and islands? In that case, split it up in several polygons. Hope this helps. Anyway, it should not need to take this long. Raymond On 09-09-2022 16:28, Dennis Burgess via Qgis-user wrote: > How can I speed up Intersects? Right now I have a single geometry > file, but then have about 2 million points that I need to understand > what intersects that geometry. Right now its taking 3 days? > QGIS-bin.exe is only using 11% CPU.. ☹ > > Dennis > > > ___ > Qgis-user mailing list > Qgis-user@lists.osgeo.org<mailto:Qgis-user@lists.osgeo.org> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org<mailto:Qgis-user@lists.osgeo.org> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org<mailto:Qgis-user@lists.osgeo.org> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] Hire a QGIS Contractor
Looking for a contractor that can either speak with us and/or exchange emails in English for QGIS . Anyone please hit me off-list [LTI-Full_175px] Dennis Burgess Mikrotik : Trainer, Network Associate, Routing Engineer, Wireless Engineer, Traffic Control Engineer, Inter-Networking Engineer, Security Engineer, Enterprise Wireless Engineer Hurricane Electric: IPv6 Sage Level Cambium: ePMP Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Create Wireless Coverage’s with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net How did we do today? [Gold Star]<https://app.customerthermometer.com/?template=log_feedback=5badbac1_data=dGVtcGVyYXR1cmVfaWQ9MSZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Green Light]<https://app.customerthermometer.com/?template=log_feedback=675abe04_data=dGVtcGVyYXR1cmVfaWQ9MiZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Yellow Light]<https://app.customerthermometer.com/?template=log_feedback=e42b48a5_data=dGVtcGVyYXR1cmVfaWQ9MyZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===>[Red Light]<https://app.customerthermometer.com/?template=log_feedback=ecaadcd3_data=dGVtcGVyYXR1cmVfaWQ9NCZ0aGVybW9tZXRlcl9pZD0xMTM1NjYmbnBzX3JhdGluZz0tMQ===Anonymous=Dennis=Burgess===> ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] Faster Intersects
In the format, are you talking about the shape or the CSV with 2million lines? Dennis Burgess Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net Create Wireless Coverage’s with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net -Original Message- From: Qgis-user On Behalf Of Raymond Nijssen via Qgis-user Sent: Friday, September 9, 2022 11:11 AM To: qgis-user@lists.osgeo.org Subject: Re: [Qgis-user] Faster Intersects Some things to check: * Is your data local? * Is it in a fast file format? (for example .gpkg or .shp and not .csv or .geojson) * Does the data have a spatial index? * Is the geometry very complicated, like a huge multi polygon with thousands of vertices and islands? In that case, split it up in several polygons. Hope this helps. Anyway, it should not need to take this long. Raymond On 09-09-2022 16:28, Dennis Burgess via Qgis-user wrote: > How can I speed up Intersects? Right now I have a single geometry > file, but then have about 2 million points that I need to understand > what intersects that geometry. Right now its taking 3 days? > QGIS-bin.exe is only using 11% CPU.. ☹ > > Dennis > > > ___ > Qgis-user mailing list > Qgis-user@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Re: [Qgis-user] Faster Intersects
Is there a postGIS or other community, I need an automated method to move data from MSSQL to POSTGIS for processing? And will postgis run faster than MSSQL? Will it use all cores? [LTI-Full_175px] Dennis Burgess Author of "Learn RouterOS- Second Edition” Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net<http://www.linktechs.net/> Create Wireless Coverage’s with www.towercoverage.com Need MikroTik Cloud Management: https://cloud.linktechs.net From: Alexandre Neto Sent: Friday, September 9, 2022 11:19 AM To: Dennis Burgess Cc: QGIS User Subject: Re: [Qgis-user] Faster Intersects I don't know what is your data format, but make sure you have spatial index in it. For things with millions of points I would move the data into PostGIS for definitely faster processing. Alexandre Neto User Support A sexta, 9/09/2022, 15:50, Dennis Burgess via Qgis-user mailto:qgis-user@lists.osgeo.org>> escreveu: How can I speed up Intersects? Right now I have a single geometry file, but then have about 2 million points that I need to understand what intersects that geometry. Right now its taking 3 days? QGIS-bin.exe is only using 11% CPU.. ☹ Dennis ___ Qgis-user mailing list Qgis-user@lists.osgeo.org<mailto:Qgis-user@lists.osgeo.org> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] Faster Intersects
How can I speed up Intersects? Right now I have a single geometry file, but then have about 2 million points that I need to understand what intersects that geometry. Right now its taking 3 days? QGIS-bin.exe is only using 11% CPU.. ☹ Dennis ___ Qgis-user mailing list Qgis-user@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
[Qgis-user] qgis 2.8 crash on sql spacial add layer
I have googled around a bit, but can't figure this one out. Have a very simple table, id, geom and state code Multi-poly in the geom field. All I really want to do is display it and export it to a KML, but as soon as I add the table to qgis is crashes.. any suggestions Thanks, Dennis ___ Qgis-user mailing list Qgis-user@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-user
