Re: [Qgis-user] Optimize QGIS Command_Line

[Frank Broniewski via QGIS-User]

Sorry, but I am out of ideas, I simply do not know enough about MSSQL to 
provide some insight. But I see, you are using a stored procedure instead of 
referring a table as source directly. Are you sure that this is not the 
culprit? It might work fine inside MSSQL but might pose problems when facing 
with the outside.

Just to rule out the file format, did you try writing to Shapefile or some 
other format to test wether it is a problem with the GeoPackage format itself 
or maybe the ogr2ogr driver needs some tweaking?

What does querying with ogrinfo on your stored procedure give you as an answer? 
Maybe there is something in there for information as well ... missing spatial 
index on your side?


Von: Dennis Burgess 
Gesendet: Samstag, 10. Februar 2024 22:36
An: Frank Broniewski ; QGIS-User@lists.osgeo.org 

Betreff: Re: [Qgis-user] Optimize QGIS Command_Line


Originally its in MSSQL, we have done a makevalid() on all lines before we use 
OGR2OGR to export it to a GeoPackage.  That file is on the local NVME drive.
We did try setting the spatial index to no, this had very little difference in 
time.


Dennis Burgess





From: Frank Broniewski 
Sent: Wednesday, February 7, 2024 12:35:52 AM
To: QGIS-User@lists.osgeo.org; Dennis Burgess
Subject: AW: [Qgis-user] Optimize QGIS Command_Line

Hi Dennis,

concerning your first command, is your GeoPackage stored locally or on a 
network drive? Writing SQLite/GepPackage databases onto a network drive can 
have a severe performance impact. Another idea, did you try setting -lco 
SPATIAL_INDEX=no and creating the index after the import? This might speed up 
your import as well.

This page https://gdal.org/drivers/vector/sqlite.html  has some more insights 
on performance, but maybe you know it already ...

Concerning your second command, without knowing how many lines / polygons you 
want to dissolve here (seems like a lot looking at the file size) it looks 
pretty normal to me. Keep in mind that dissolving is a rather cost intensive 
process, when not dissolving by attribute. As every geometry has to be checked 
against all the others if they can be dissolved spatially - spatial indexes 
aside.
I do not know the internals of the algorithm itself, but I would guess it is 
not well suited for a threaded / multi process approach and it just runs on one 
core of your CPU doing all the work - like most GIS tools do.

A speedier approach could be to spatially divide your data into smaller parts, 
do the dissolve on each part to reduce the number of features, and then finally 
dissolve the results into the final one ...




Von: QGIS-User  im Auftrag von Dennis 
Burgess via QGIS-User 
Gesendet: Dienstag, 6. Februar 2024 15:23
An: QGIS-User@lists.osgeo.org 
Betreff: Re: [Qgis-user] Optimize QGIS Command_Line


ogr2ogr -f gpkg geom_multi_38.gpkg 
"MSSQL:server=xx;database=x;uid=x;pwd=;MultipleActiveResultSets=true;"
 -sql "EXEC xx 42039, 38" -overwrite  -gt 100 -lco SPATIAL_INDEX=yes  
OGR_SQLITE_CACHE=10240MB OGR_SQLITE_SYNCHRONOUS=OFF



This one takes around 30 min to export to gpkg file, if I run the same exec 
command on sql it takes less than 40 seconds to pull down ..  these are 
geometry lines.   Around 940 lines.



The second one





"c:\program files\qgis 3.26.2\apps\qgis\bin\qgis_process" run native:dissolve 
--INPUT=geom_multi_55096.gpkg --OUTPUT=geom_multi_diss_38.gpkg



This would e one of them.  Last job took 49 minutes, orginal input is around 
1.2 gig.



Dedicated platform for processing: 72 cores, 32 gig of RAM, running on NVME 
disk.

SQL 2019 Enterprise: 80 cores, 512gig ram, NVME disks.





From: Johannes Kröger (WhereGroup) 
Sent: Tuesday, February 6, 2024 2:01 AM
To: Dennis Burgess ; QGIS-User@lists.osgeo.org
Subject: Re: [Qgis-user] Optimize QGIS Command_Line



Hey Dennis,

can you share those command lines (maybe with sensitive information stripped)? 
Otherwise it is hard to guess what the issue might be.

What does your CPU, RAM and I/O utilisation look like during those long running 
processes? Same for the MSSQL database server host system.

Cheers, Hannes

On 05.02.24 14:46, Dennis Burgess via QGIS-User wrote:

I have two  command lines, one reads from my MSSQL database and creates a 
GeoPackage on the local disk.  This takes WAY too long, just to create it , it 
uses 5% or less CPU, and is SUPER slow. Looking for methods to speed this up.  
The data is around 1.2 to 1.4 gig of pop, but it should not take 50 min to 
create.



Also I am doing a dissolve on this data, once its on the disk, this also takes 
quite a while, even though when I do it from QGIS it takes like 10 min vs an 
hour or more on the computer with the command line.



I’m sure I am doing something wrong on both of these. Just trying to understand 
what it would be.



Thanks for any assistance !

Dennis

Re: [Qgis-user] Creating Wedge buffers automatically

[krishna Ayyala via QGIS-User]

Thank You Andrea. This worked for me. Thanks a lot for your help.

Regards.

On Sat, Feb 10, 2024 at 12:07 AM Andrea Giudiceandrea via QGIS-User <
qgis-user@lists.osgeo.org> wrote:

> > krishna Ayyala ayyalakrishna at gmail.com
> > Fri Feb 9 16:26:48 PST 2024
> > I saw the "Create wedge buffers tool"
> > in QGIS but that does not work for me. The reason is; in that tool we
> have
> > to manually enter the azimuth, wedge width and radius. Is there a tool
> that
> > can automatically recognize the azimuth and radius (Dist) and create
> wedge
> > buffers as shown below.
>
> Hi krishna Ayyala,
> if all the needed data values are stored in the attributes of each
> feature, then you can use the "data defined override" functionality
> described in the QGIS user manual at
>
> https://docs.qgis.org/3.34/en/docs/user_manual/introduction/general_tools.html#data-defined-override-setup
>
> Best regards.
>
> Andrea
> ___
> QGIS-User mailing list
> QGIS-User@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Batch Processing Tool for Wedge Buffers

[DelazJ via QGIS-User]

Hi
Krishna Ayyala, are you sure the batch processing is what you need? You mention 
you have a single file. In your original message "creating wedge buffers 
automatically", Andrea suggested to use the algorithm in the "default" mode, 
setting the fields as arguments through the data-defined button next to each. 
No feedback yet. Have you tried his suggestions?

Regards,
Harrissou

Le 11 février 2024 17:12:57 GMT+01:00, Ujaval Gandhi via QGIS-User 
 a écrit :
>___
>QGIS-User mailing list
>QGIS-User@lists.osgeo.org
>List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Batch Processing Tool for Wedge Buffers

[Ujaval Gandhi via QGIS-User]

Should be possible. Use Autofill -> Calculate by _expression_ can build an _expression_ that takes the Layer (you have access to the current layer variable in _expression_ builder) and the Row number (rownumber variable) and get the attribute value of 'Azimuth' field. ---Ujaval GandhiSpatial Thoughtswww.spatialthoughts.comOn Sun, Feb 11, 2024 at 9:32 PM krishna Ayyala  wrote:Ujaval,Thanks for the reply. When I add  generate_series _expression_ and click ok, only one more row is getting added to the batch process tool. i.e.90 degree azimuth got added. 180 and 270 were not added. Is there a way that, the batch tool can read directly from the attribute table and generate the wedge buffers? The reason is; the example that I gave has only four features.But my original file has 100 features. It will be difficult to enter all 100 values manually. If the tool can pull these values from the attribute table and generate wedge buffers that will be free of errors.On Sun, Feb 11, 2024 at 5:19 AM Ujaval Gandhi  wrote:click 'Autofill' -> Add Values by _expression_ and use _expression_ generate_series(90, 270, 90)You will get 4 layers, one for each azimuth value. Use 'Merge Vector Layers' tool to merge them into a single layer.---Ujaval GandhiSpatial Thoughtswww.spatialthoughts.comOn Sun, Feb 11, 2024 at 3:25 AM krishna Ayyala via QGIS-User  wrote:Hello, I have opened Batch Processing tool for wedge buffers. In this, how do I use the Auto fill function such that, I get the four rows with azimuths 0,90,180,270 without me having to manually input the azimuth values for the attribute table as shown below.

___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Batch Processing Tool for Wedge Buffers

[krishna Ayyala via QGIS-User]

Ujaval,
Thanks for the reply. When I add  generate_series expression and click ok,
only one more row is getting added to the batch process tool. i.e.90 degree
azimuth got added. 180 and 270 were not added. Is there a way that, the
batch tool can read directly from the attribute table and generate the
wedge buffers? The reason is; the example that I gave has only four
features.But my original file has 100 features. It will be difficult to
enter all 100 values manually. If the tool can pull these values from the
attribute table and generate wedge buffers that will be free of errors.

[image: image.png]

[image: image.png]

[image: image.png]


On Sun, Feb 11, 2024 at 5:19 AM Ujaval Gandhi 
wrote:

> click 'Autofill' -> Add Values by Expression and use
> expression generate_series(90, 270, 90)
>
> You will get 4 layers, one for each azimuth value. Use 'Merge Vector
> Layers' tool to merge them into a single layer.
> ---
> Ujaval Gandhi
> Spatial Thoughts
> www.spatialthoughts.com
> 
>
>
>
> On Sun, Feb 11, 2024 at 3:25 AM krishna Ayyala via QGIS-User <
> qgis-user@lists.osgeo.org> wrote:
>
>> Hello,
>> I have opened Batch Processing tool for wedge buffers. In this, how do I
>> use the Auto fill function such that, I get the four rows with azimuths
>> 0,90,180,270 without me having to manually input the azimuth values for the
>> attribute table as shown below.
>> [image: image.png]
>>
>> [image: image.png]
>>
>>
>>> ___
>> QGIS-User mailing list
>> QGIS-User@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>
___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] Batch Processing Tool for Wedge Buffers

[Ujaval Gandhi via QGIS-User]

click 'Autofill' -> Add Values by _expression_ and use _expression_ generate_series(90, 270, 90)You will get 4 layers, one for each azimuth value. Use 'Merge Vector Layers' tool to merge them into a single layer.---Ujaval GandhiSpatial Thoughtswww.spatialthoughts.comOn Sun, Feb 11, 2024 at 3:25 AM krishna Ayyala via QGIS-User  wrote:Hello, I have opened Batch Processing tool for wedge buffers. In this, how do I use the Auto fill function such that, I get the four rows with azimuths 0,90,180,270 without me having to manually input the azimuth values for the attribute table as shown below.

___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
___
QGIS-User mailing list
QGIS-User@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [QGIS-it-user] Digest di QGIS-it-user, Volume 99, Numero 17

[Carlo A. Nicolini via QGIS-it-user]

Buongiorno a tutti,

per quanto estraneo allo sviluppo, da utente finale concordo con quanto 
detto da Tony.


Ahime


Saluti

Carlo A.

Il 10/02/24 21:00, qgis-it-user-requ...@lists.osgeo.org ha scritto:

Bisognerebbe invece fare una campagna per far si che lo Stato, le Regioni e
i Comuni finanzino il progetto e i plug in che vengono usati
quotidianamente per sviluppare piani urbanistici e altri aspetti tecnici ma
con la filosofia "Qgis è gratis".

Stessa cosa vale per le aziende che utilizzano il software e i vari plug in.___
QGIS-it-user mailing list
QGIS-it-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/qgis-it-user