Re: [QGIS-Developer] How to speed up rendering

[afernandez via QGIS-Developer]

Hello Hannes, Andreas and Ian,
Thank you for the links. They have been really useful for understanding the 
overviews/pyramids concept. I'm still experimenting but was able to decrease 
the file size.
Thanks,
Arturo
Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
Sorry Arturo,
I totally missed that Martin had already recommended it and that you said you 
didn't know about it. I didn't mean to come across pushy or anything :))
Andreas' links are great to see how to create them. Here is a nice post that 
explains the concept: 
https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/
 

As you can see there you probably also want to created "tiled" TIFFs, where the 
data internal to the TIFF is arranged in a way that fits spatial queries better.
If you can, simply specify that you want "cloud-optimized geotiffs", those have 
all these optimisations and are also "typical spatial analysis optimized 
geotiffs".
Cheers, Hannes
Am 30.05.23 um 14:37 schrieb afernandez via QGIS-Developer:
Hello Hannes,
Sorry to say that I'm at a loss here. I had not previously heard of 'overviews' 
and am having a hard time finding much information. I couldn't find anything in 
the PyQGIS (v3.28) developer cookbook, which is my main reference in this QGIS 
journey. The found information is at 
https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
 

 but didn't exactly clarify how to implement overviews with a script or why 
this would sidestep reading the file several times. I don't quite understand 
this last point either, as the file is large but the size of the band (after 
everything is prepped and before rendering) is relatively small, and should fit 
amply into the available memory.
Thanks,
Arturo
Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
Hi Arturo,
make sure you create overviews on your big rasters or QGIS will re-read the 
whole file all the time to render an appropriate representation in the map.
Cheers, Hannes
Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
Hello Martin,
I'm trying to understand what you're saying but my background is in numerical 
methods where tasks are split into threads for acceleration purposes. If a 
single layer is rendered by a single core (as you write and I was observing), 
how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be handled? Wouldn't it 
take too long? (However, you want to define 'too long').
As far as more details about the process, the tensor 'var' (n x m x index where 
I have tried different combinations but let's say n x m is 100 x 200) of 
floating point numbers is manipulated according to the following pseudo-code:
# Initial manipulations
dims = var.dimensions
shape = var.shape
driver_name = 'GTIFF'
driver = gdal.GetDriverByName(driver_name)
np_dtype = var.dtype
type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
gdal_ds = driver.Create(_my_path_, cols, rows, 1, gdal.GDT_UInt16)
gdal_ds.SetProjection(_my_projection_)
gdal_ds.SetGeoTransform(_my_transformation_)
# Creation of the bands and scaled matrix
band = gdal_ds.GetRasterBand(1)
data = var[_chosen_index_]
data = ma.getdata(data)
data_scaled = np.interp(data, (data.min(), data.max()), (0, 255))
data_scaled2 = data_scaled.astype(int) # This is to rescale into integers so 
that it can color the layer
# *** Lines to set up the color palette ***
# Write the array to band once everything has been rescaled
band.WriteArray(data_scaled2)
gdal_ds.FlushCache()
This procedure seems to work well and takes a reasonable amount of time. Then, 
the time is consumed when doing the rendering itself. I'm not entirely sure 
what you mean by 'sub-optimal raster format' or 'thre are overviews missing' so 
I cannot address them specifically.
Thanks,
Arturo
Martin Dobias wrote:
Hi
On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer 
mailto:qgis-developer@lists.osgeo.org> > wrote:
Hello,
One of my project would greatly benefit from a much faster rendering. After 
some testing, the bulk (>99%) of the time is spent at rendering the layer with 
QgsRasterLayer(...,...). What I have also noticed is that rendering time does 
not change with the number of cores, which suggests that the renderer might be 
using a single thread. I checked the API documentation but couldn't find 
anything about multithreading or some other procedure to accelerate rendering. 
I was wondering if anyone has any suggestion or has faced a similar difficulty.
Multi-threaded rendering is done at the granularity of whole map layers, so a 
single layer is rendered by a single CPU core. Therefore if you have just one 
layer that is slow to render, the multi-threading is not going to help. It is 
best to do some testing how you 

Re: [QGIS-Developer] How to speed up rendering

[Ian Turton via QGIS-Developer]

A good introduction to fast rasters is Paul Ramsey's GeoTiff compression
for dummies blog post -
https://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html
it takes you through all the steps needed using gdal.

Ian

On Wed, 31 May 2023 at 08:47, Johannes Kröger (WhereGroup) via
QGIS-Developer  wrote:

> Sorry Arturo,
>
> I totally missed that Martin had already recommended it and that you said
> you didn't know about it. I didn't mean to come across pushy or anything :))
>
> Andreas' links are great to see how to create them. Here is a nice post
> that explains the concept:
> https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/
>
> As you can see there you probably also want to created "tiled" TIFFs,
> where the data internal to the TIFF is arranged in a way that fits spatial
> queries better.
>
> If you can, simply specify that you want "cloud-optimized geotiffs", those
> have all these optimisations and are also "typical spatial analysis
> optimized geotiffs".
>
> Cheers, Hannes
> Am 30.05.23 um 14:37 schrieb afernandez via QGIS-Developer:
>
> Hello Hannes,
> Sorry to say that I'm at a loss here. I had not previously heard of
> 'overviews' and am having a hard time finding much information. I couldn't
> find anything in the PyQGIS (v3.28) developer cookbook, which is my main
> reference in this QGIS journey. The found information is at
> https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
> but didn't exactly clarify how to implement overviews with a script or why
> this would sidestep reading the file several times. I don't quite
> understand this last point either, as the file is large but the size of the
> band (after everything is prepped and before rendering) is relatively
> small, and should fit amply into the available memory.
> Thanks,
> Arturo
>
>
> Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
>
>
> Hi Arturo,
>
> make sure you create overviews on your big rasters or QGIS will re-read
> the whole file all the time to render an appropriate representation in the
> map.
>
> Cheers, Hannes
> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>
> Hello Martin,
> I'm trying to understand what you're saying but my background is in
> numerical methods where tasks are split into threads for acceleration
> purposes. If a single layer is rendered by a single core (as you write and
> I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be
> handled? Wouldn't it take too long? (However, you want to define 'too
> long').
> As far as more details about the process, the tensor 'var' (n x m x index
> where I have tried different combinations but let's say n x m is 100 x 200)
> of floating point numbers is manipulated according to the following
> pseudo-code:
> # Initial manipulations
> dims = var.dimensions
> shape = var.shape
> driver_name = 'GTIFF'
> driver = gdal.GetDriverByName(driver_name)
> np_dtype = var.dtype
> type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
> gdal.GDT_UInt16)
> gdal_ds.SetProjection(_my_projection_)
> gdal_ds.SetGeoTransform(_my_transformation_)
> # Creation of the bands and scaled matrix
> band = gdal_ds.GetRasterBand(1)
> data = var[_chosen_index_]
> data = ma.getdata(data)
> data_scaled =  np.interp(data, (data.min(), data.max()), (0,
> 255))
> data_scaled2 = data_scaled.astype(int) # This is to rescale
> into integers so that it can color the layer
> # *** Lines to set up the color palette ***
> # Write the array to band once everything has been rescaled
> band.WriteArray(data_scaled2)
> gdal_ds.FlushCache()
>
> This procedure seems to work well and takes a reasonable amount of time.
> Then, the time is consumed when doing the rendering itself. I'm not
> entirely sure what you mean by 'sub-optimal raster format' or 'thre are
> overviews missing' so I cannot address them specifically.
> Thanks,
> Arturo
>
>
>
>
> Martin Dobias wrote:
>
>
> Hi
>
> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer <
> qgis-developer@lists.osgeo.org> wrote:
>
>> Hello,
>> One of my project would greatly benefit from a much faster rendering.
>> After some testing, the bulk (>99%) of the time is spent at rendering the
>> layer with QgsRasterLayer(...,...). What I have also noticed is that
>> rendering time does not change with the number of cores, which suggests
>> that the renderer might be using a single thread. I checked the API
>> documentation but couldn't find anything about multithreading or some other
>> procedure to accelerate rendering. I was wondering if anyone has any
>> suggestion or has faced a similar difficulty.
>>
>
> Multi-threaded rendering is done at the 

Re: [QGIS-Developer] How to speed up rendering

[WhereGroup]

Sorry Arturo,

I totally missed that Martin had already recommended it and that you 
said you didn't know about it. I didn't mean to come across pushy or 
anything :))


Andreas' links are great to see how to create them. Here is a nice post 
that explains the concept: 
https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/


As you can see there you probably also want to created "tiled" TIFFs, 
where the data internal to the TIFF is arranged in a way that fits 
spatial queries better.


If you can, simply specify that you want "cloud-optimized geotiffs", 
those have all these optimisations and are also "typical spatial 
analysis optimized geotiffs".


Cheers, Hannes

Am 30.05.23 um 14:37 schrieb afernandez via QGIS-Developer:

Hello Hannes,
Sorry to say that I'm at a loss here. I had not previously heard of 
'overviews' and am having a hard time finding much information. I 
couldn't find anything in the PyQGIS (v3.28) developer cookbook, which 
is my main reference in this QGIS journey. The found information is at 
https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html 
but didn't exactly clarify how to implement overviews with a script or 
why this would sidestep reading the file several times. I don't quite 
understand this last point either, as the file is large but the size 
of the band (after everything is prepped and before rendering) is 
relatively small, and should fit amply into the available memory.

Thanks,
Arturo


Johannes Kröger (WhereGroup) via QGIS-Developer wrote:



Hi Arturo,

make sure you create overviews on your big rasters or QGIS will 
re-read the whole file all the time to render an appropriate 
representation in the map.


Cheers, Hannes

Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:

Hello Martin,
I'm trying to understand what you're saying but my background is in 
numerical methods where tasks are split into threads for 
acceleration purposes. If a single layer is rendered by a single 
core (as you write and I was observing), how would a 1,000 x 1,000 
(or a 5,000 x 5,000) layer be handled? Wouldn't it take too long? 
(However, you want to define 'too long').
As far as more details about the process, the tensor 'var' (n x m x 
index where I have tried different combinations but let's say n x m 
is 100 x 200) of floating point numbers is manipulated according to 
the following pseudo-code:

# Initial manipulations
            dims = var.dimensions
            shape = var.shape
            driver_name = 'GTIFF'
            driver = gdal.GetDriverByName(driver_name)
            np_dtype = var.dtype
            type_code = 
gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
            gdal_ds = driver.Create(_my_path_, cols, rows, 1, 
gdal.GDT_UInt16)

gdal_ds.SetProjection(_my_projection_)
gdal_ds.SetGeoTransform(_my_transformation_)
# Creation of the bands and scaled matrix
            band = gdal_ds.GetRasterBand(1)
            data = var[_chosen_index_]
            data = ma.getdata(data)
            data_scaled = np.interp(data, (data.min(), data.max()), 
(0, 255))
            data_scaled2 = data_scaled.astype(int) # This is to 
rescale into integers so that it can color the layer

# *** Lines to set up the color palette ***
# Write the array to band once everything has been rescaled
band.WriteArray(data_scaled2)
            gdal_ds.FlushCache()

This procedure seems to work well and takes a reasonable amount of 
time. Then, the time is consumed when doing the rendering itself. 
I'm not entirely sure what you mean by 'sub-optimal raster format' 
or 'thre are overviews missing' so I cannot address them specifically.

Thanks,
Arturo




Martin Dobias wrote:



Hi

On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer 
 wrote:


Hello,
One of my project would greatly benefit from a much faster
rendering. After some testing, the bulk (>99%) of the time is
spent at rendering the layer with QgsRasterLayer(...,...). What
I have also noticed is that rendering time does not change with
the number of cores, which suggests that the renderer might be
using a single thread. I checked the API documentation but
couldn't find anything about multithreading or some other
procedure to accelerate rendering. I was wondering if anyone
has any suggestion or has faced a similar difficulty.


Multi-threaded rendering is done at the granularity of whole map 
layers, so a single layer is rendered by a single CPU core. 
Therefore if you have just one layer that is slow to render, the 
multi-threading is not going to help. It is best to do some testing 
how you can improve the layer's rendering - for example, it could 
be that you are using some sub-optimal raster format, or there are 
overviews missing, or something else... one would need more details.


Regards
Martin



___
QGIS-Developer mailing 

Re: [QGIS-Developer] How to speed up rendering

[Andreas Neumann via QGIS-Developer]

Hi,

See
https://docs.qgis.org/3.28/en/docs/user_manual/working_with_raster/raster_properties.html#pyramids-properties
(for a QGIS based solution)
or https://gdal.org/programs/gdaladdo.html (for a command line or python
version).

Andreas

On Tue, 30 May 2023 at 16:00, afernandez via QGIS-Developer <
qgis-developer@lists.osgeo.org> wrote:

> Hello Hannes,
> Sorry to say that I'm at a loss here. I had not previously heard of
> 'overviews' and am having a hard time finding much information. I couldn't
> find anything in the PyQGIS (v3.28) developer cookbook, which is my main
> reference in this QGIS journey. The found information is at
> https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
> but didn't exactly clarify how to implement overviews with a script or why
> this would sidestep reading the file several times. I don't quite
> understand this last point either, as the file is large but the size of the
> band (after everything is prepped and before rendering) is relatively
> small, and should fit amply into the available memory.
> Thanks,
> Arturo
>
>
> Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
>
>
> Hi Arturo,
>
> make sure you create overviews on your big rasters or QGIS will re-read
> the whole file all the time to render an appropriate representation in the
> map.
>
> Cheers, Hannes
> Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
>
> Hello Martin,
> I'm trying to understand what you're saying but my background is in
> numerical methods where tasks are split into threads for acceleration
> purposes. If a single layer is rendered by a single core (as you write and
> I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be
> handled? Wouldn't it take too long? (However, you want to define 'too
> long').
> As far as more details about the process, the tensor 'var' (n x m x index
> where I have tried different combinations but let's say n x m is 100 x 200)
> of floating point numbers is manipulated according to the following
> pseudo-code:
> # Initial manipulations
> dims = var.dimensions
> shape = var.shape
> driver_name = 'GTIFF'
> driver = gdal.GetDriverByName(driver_name)
> np_dtype = var.dtype
> type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
> gdal_ds = driver.Create(_my_path_, cols, rows, 1,
> gdal.GDT_UInt16)
> gdal_ds.SetProjection(_my_projection_)
> gdal_ds.SetGeoTransform(_my_transformation_)
> # Creation of the bands and scaled matrix
> band = gdal_ds.GetRasterBand(1)
> data = var[_chosen_index_]
> data = ma.getdata(data)
> data_scaled =  np.interp(data, (data.min(), data.max()), (0,
> 255))
> data_scaled2 = data_scaled.astype(int) # This is to rescale
> into integers so that it can color the layer
> # *** Lines to set up the color palette ***
> # Write the array to band once everything has been rescaled
> band.WriteArray(data_scaled2)
> gdal_ds.FlushCache()
>
> This procedure seems to work well and takes a reasonable amount of time.
> Then, the time is consumed when doing the rendering itself. I'm not
> entirely sure what you mean by 'sub-optimal raster format' or 'thre are
> overviews missing' so I cannot address them specifically.
> Thanks,
> Arturo
>
>
>
>
> Martin Dobias wrote:
>
>
> Hi
>
> On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer <
> qgis-developer@lists.osgeo.org> wrote:
>
>> Hello,
>> One of my project would greatly benefit from a much faster rendering.
>> After some testing, the bulk (>99%) of the time is spent at rendering the
>> layer with QgsRasterLayer(...,...). What I have also noticed is that
>> rendering time does not change with the number of cores, which suggests
>> that the renderer might be using a single thread. I checked the API
>> documentation but couldn't find anything about multithreading or some other
>> procedure to accelerate rendering. I was wondering if anyone has any
>> suggestion or has faced a similar difficulty.
>>
>
> Multi-threaded rendering is done at the granularity of whole map layers,
> so a single layer is rendered by a single CPU core. Therefore if you have
> just one layer that is slow to render, the multi-threading is not going to
> help. It is best to do some testing how you can improve the layer's
> rendering - for example, it could be that you are using some sub-optimal
> raster format, or there are overviews missing, or something else... one
> would need more details.
>
> Regards
> Martin
>
>
> ___
> QGIS-Developer mailing listqgis-develo...@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
> Johannes Kröger / GIS-Entwickler/-Berater
>
> 

Re: [QGIS-Developer] How to speed up rendering

[afernandez via QGIS-Developer]

Hello Hannes,
Sorry to say that I'm at a loss here. I had not previously heard of 'overviews' 
and am having a hard time finding much information. I couldn't find anything in 
the PyQGIS (v3.28) developer cookbook, which is my main reference in this QGIS 
journey. The found information is at 
https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
 but didn't exactly clarify how to implement overviews with a script or why 
this would sidestep reading the file several times. I don't quite understand 
this last point either, as the file is large but the size of the band (after 
everything is prepped and before rendering) is relatively small, and should fit 
amply into the available memory.
Thanks,
Arturo
Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
Hi Arturo,
make sure you create overviews on your big rasters or QGIS will re-read the 
whole file all the time to render an appropriate representation in the map.
Cheers, Hannes
Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
Hello Martin,
I'm trying to understand what you're saying but my background is in numerical 
methods where tasks are split into threads for acceleration purposes. If a 
single layer is rendered by a single core (as you write and I was observing), 
how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be handled? Wouldn't it 
take too long? (However, you want to define 'too long').
As far as more details about the process, the tensor 'var' (n x m x index where 
I have tried different combinations but let's say n x m is 100 x 200) of 
floating point numbers is manipulated according to the following pseudo-code:
# Initial manipulations
dims = var.dimensions
shape = var.shape
driver_name = 'GTIFF'
driver = gdal.GetDriverByName(driver_name)
np_dtype = var.dtype
type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
gdal_ds = driver.Create(_my_path_, cols, rows, 1, gdal.GDT_UInt16)
gdal_ds.SetProjection(_my_projection_)
gdal_ds.SetGeoTransform(_my_transformation_)
# Creation of the bands and scaled matrix
band = gdal_ds.GetRasterBand(1)
data = var[_chosen_index_]
data = ma.getdata(data)
data_scaled = np.interp(data, (data.min(), data.max()), (0, 255))
data_scaled2 = data_scaled.astype(int) # This is to rescale into integers so 
that it can color the layer
# *** Lines to set up the color palette ***
# Write the array to band once everything has been rescaled
band.WriteArray(data_scaled2)
gdal_ds.FlushCache()
This procedure seems to work well and takes a reasonable amount of time. Then, 
the time is consumed when doing the rendering itself. I'm not entirely sure 
what you mean by 'sub-optimal raster format' or 'thre are overviews missing' so 
I cannot address them specifically.
Thanks,
Arturo
Martin Dobias wrote:
Hi
On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer 
mailto:qgis-developer@lists.osgeo.org> > wrote:
Hello,
One of my project would greatly benefit from a much faster rendering. After 
some testing, the bulk (>99%) of the time is spent at rendering the layer with 
QgsRasterLayer(...,...). What I have also noticed is that rendering time does 
not change with the number of cores, which suggests that the renderer might be 
using a single thread. I checked the API documentation but couldn't find 
anything about multithreading or some other procedure to accelerate rendering. 
I was wondering if anyone has any suggestion or has faced a similar difficulty.
Multi-threaded rendering is done at the granularity of whole map layers, so a 
single layer is rendered by a single CPU core. Therefore if you have just one 
layer that is slow to render, the multi-threading is not going to help. It is 
best to do some testing how you can improve the layer's rendering - for 
example, it could be that you are using some sub-optimal raster format, or 
there are overviews missing, or something else... one would need more details.
Regards
Martin
___ QGIS-Developer mailing list 
QGIS-Developer@lists.osgeo.org  List 
info: https://lists.osgeo.org/mailman/listinfo/qgis-developer 
 Unsubscribe: 
https://lists.osgeo.org/mailman/listinfo/qgis-developer 
 -- Johannes Kröger / 
GIS-Entwickler/-Berater - Aufwind 
durch Wissen! Web-Seminare und Online-Schulungen bei der www.foss-academy.com 
 - 
WhereGroup GmbH c/o KK03 GmbH Lange Reihe 29 20099 Hamburg Germany Tel: +49 
(0)228 / 90 90 38 - 36 Fax: +49 (0)228 / 90 90 38 - 11 
johannes.kroe...@wheregroup.com  
www.wheregroup.com  Geschäftsführer: Olaf Knopp, 
Peter Stamm Amtsgericht Bonn, HRB 9885 ---
Hi Arturo,
make sure you create overviews on 

Re: [QGIS-Developer] How to speed up rendering

[WhereGroup]

Hi Arturo,

make sure you create overviews on your big rasters or QGIS will re-read 
the whole file all the time to render an appropriate representation in 
the map.


Cheers, Hannes

Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:

Hello Martin,
I'm trying to understand what you're saying but my background is in 
numerical methods where tasks are split into threads for acceleration 
purposes. If a single layer is rendered by a single core (as you write 
and I was observing), how would a 1,000 x 1,000 (or a 5,000 x 5,000) 
layer be handled? Wouldn't it take too long? (However, you want to 
define 'too long').
As far as more details about the process, the tensor 'var' (n x m x 
index where I have tried different combinations but let's say n x m is 
100 x 200) of floating point numbers is manipulated according to the 
following pseudo-code:

# Initial manipulations
            dims = var.dimensions
            shape = var.shape
            driver_name = 'GTIFF'
            driver = gdal.GetDriverByName(driver_name)
            np_dtype = var.dtype
            type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
            gdal_ds = driver.Create(_my_path_, cols, rows, 1, 
gdal.GDT_UInt16)

gdal_ds.SetProjection(_my_projection_)
gdal_ds.SetGeoTransform(_my_transformation_)
# Creation of the bands and scaled matrix
            band = gdal_ds.GetRasterBand(1)
            data = var[_chosen_index_]
            data = ma.getdata(data)
            data_scaled =  np.interp(data, (data.min(), data.max()), 
(0, 255))
            data_scaled2 = data_scaled.astype(int) # This is to 
rescale into integers so that it can color the layer

# *** Lines to set up the color palette ***
# Write the array to band once everything has been rescaled
            band.WriteArray(data_scaled2)
            gdal_ds.FlushCache()

This procedure seems to work well and takes a reasonable amount of 
time. Then, the time is consumed when doing the rendering itself. I'm 
not entirely sure what you mean by 'sub-optimal raster format' or 
'thre are overviews missing' so I cannot address them specifically.

Thanks,
Arturo




Martin Dobias wrote:



Hi

On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer 
 wrote:


Hello,
One of my project would greatly benefit from a much faster
rendering. After some testing, the bulk (>99%) of the time is
spent at rendering the layer with QgsRasterLayer(...,...). What I
have also noticed is that rendering time does not change with the
number of cores, which suggests that the renderer might be using
a single thread. I checked the API documentation but couldn't
find anything about multithreading or some other procedure to
accelerate rendering. I was wondering if anyone has any
suggestion or has faced a similar difficulty.


Multi-threaded rendering is done at the granularity of whole map 
layers, so a single layer is rendered by a single CPU core. Therefore 
if you have just one layer that is slow to render, the 
multi-threading is not going to help. It is best to do some testing 
how you can improve the layer's rendering - for example, it could be 
that you are using some sub-optimal raster format, or there are 
overviews missing, or something else... one would need more details.


Regards
Martin



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer


--
Johannes Kröger / GIS-Entwickler/-Berater

-
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei derwww.foss-academy.com
-

WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany

Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11

johannes.kroe...@wheregroup.com
www.wheregroup.com
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
---
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] How to speed up rendering

[afernandez via QGIS-Developer]

Hello Martin,
I'm trying to understand what you're saying but my background is in numerical 
methods where tasks are split into threads for acceleration purposes. If a 
single layer is rendered by a single core (as you write and I was observing), 
how would a 1,000 x 1,000 (or a 5,000 x 5,000) layer be handled? Wouldn't it 
take too long? (However, you want to define 'too long').
As far as more details about the process, the tensor 'var' (n x m x index where 
I have tried different combinations but let's say n x m is 100 x 200) of 
floating point numbers is manipulated according to the following pseudo-code:
# Initial manipulations
dims = var.dimensions
shape = var.shape
driver_name = 'GTIFF'
driver = gdal.GetDriverByName(driver_name)
np_dtype = var.dtype
type_code = gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
gdal_ds = driver.Create(_my_path_, cols, rows, 1, gdal.GDT_UInt16)
gdal_ds.SetProjection(_my_projection_)
gdal_ds.SetGeoTransform(_my_transformation_)
# Creation of the bands and scaled matrix
band = gdal_ds.GetRasterBand(1)
data = var[_chosen_index_]
data = ma.getdata(data)
data_scaled = np.interp(data, (data.min(), data.max()), (0, 255))
data_scaled2 = data_scaled.astype(int) # This is to rescale into integers so 
that it can color the layer
# *** Lines to set up the color palette ***
# Write the array to band once everything has been rescaled
band.WriteArray(data_scaled2)
gdal_ds.FlushCache()
This procedure seems to work well and takes a reasonable amount of time. Then, 
the time is consumed when doing the rendering itself. I'm not entirely sure 
what you mean by 'sub-optimal raster format' or 'thre are overviews missing' so 
I cannot address them specifically.
Thanks,
Arturo
Martin Dobias wrote:
Hi
On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer 
mailto:qgis-developer@lists.osgeo.org> > wrote:
Hello,
One of my project would greatly benefit from a much faster rendering. After 
some testing, the bulk (>99%) of the time is spent at rendering the layer with 
QgsRasterLayer(...,...). What I have also noticed is that rendering time does 
not change with the number of cores, which suggests that the renderer might be 
using a single thread. I checked the API documentation but couldn't find 
anything about multithreading or some other procedure to accelerate rendering. 
I was wondering if anyone has any suggestion or has faced a similar difficulty.
Multi-threaded rendering is done at the granularity of whole map layers, so a 
single layer is rendered by a single CPU core. Therefore if you have just one 
layer that is slow to render, the multi-threading is not going to help. It is 
best to do some testing how you can improve the layer's rendering - for 
example, it could be that you are using some sub-optimal raster format, or 
there are overviews missing, or something else... one would need more details.
Regards
Martin
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] How to speed up rendering

[Martin Dobias via QGIS-Developer]

Hi

On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer <
qgis-developer@lists.osgeo.org> wrote:

> Hello,
> One of my project would greatly benefit from a much faster rendering.
> After some testing, the bulk (>99%) of the time is spent at rendering the
> layer with QgsRasterLayer(...,...). What I have also noticed is that
> rendering time does not change with the number of cores, which suggests
> that the renderer might be using a single thread. I checked the API
> documentation but couldn't find anything about multithreading or some other
> procedure to accelerate rendering. I was wondering if anyone has any
> suggestion or has faced a similar difficulty.
>

Multi-threaded rendering is done at the granularity of whole map layers, so
a single layer is rendered by a single CPU core. Therefore if you have just
one layer that is slow to render, the multi-threading is not going to help.
It is best to do some testing how you can improve the layer's rendering -
for example, it could be that you are using some sub-optimal raster format,
or there are overviews missing, or something else... one would need more
details.

Regards
Martin
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer