Sorry Arturo,
I totally missed that Martin had already recommended it and that you
said you didn't know about it. I didn't mean to come across pushy or
anything :))
Andreas' links are great to see how to create them. Here is a nice post
that explains the concept:
https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/
As you can see there you probably also want to created "tiled" TIFFs,
where the data internal to the TIFF is arranged in a way that fits
spatial queries better.
If you can, simply specify that you want "cloud-optimized geotiffs",
those have all these optimisations and are also "typical spatial
analysis optimized geotiffs".
Cheers, Hannes
Am 30.05.23 um 14:37 schrieb afernandez via QGIS-Developer:
Hello Hannes,
Sorry to say that I'm at a loss here. I had not previously heard of
'overviews' and am having a hard time finding much information. I
couldn't find anything in the PyQGIS (v3.28) developer cookbook, which
is my main reference in this QGIS journey. The found information is at
https://docs.qgis.org/3.28/en/docs/user_manual/processing_algs/gdal/rastermiscellaneous.html
but didn't exactly clarify how to implement overviews with a script or
why this would sidestep reading the file several times. I don't quite
understand this last point either, as the file is large but the size
of the band (after everything is prepped and before rendering) is
relatively small, and should fit amply into the available memory.
Thanks,
Arturo
Johannes Kröger (WhereGroup) via QGIS-Developer wrote:
Hi Arturo,
make sure you create overviews on your big rasters or QGIS will
re-read the whole file all the time to render an appropriate
representation in the map.
Cheers, Hannes
Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
Hello Martin,
I'm trying to understand what you're saying but my background is in
numerical methods where tasks are split into threads for
acceleration purposes. If a single layer is rendered by a single
core (as you write and I was observing), how would a 1,000 x 1,000
(or a 5,000 x 5,000) layer be handled? Wouldn't it take too long?
(However, you want to define 'too long').
As far as more details about the process, the tensor 'var' (n x m x
index where I have tried different combinations but let's say n x m
is 100 x 200) of floating point numbers is manipulated according to
the following pseudo-code:
# Initial manipulations
dims = var.dimensions
shape = var.shape
driver_name = 'GTIFF'
driver = gdal.GetDriverByName(driver_name)
np_dtype = var.dtype
type_code =
gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
gdal_ds = driver.Create(_my_path_, cols, rows, 1,
gdal.GDT_UInt16)
gdal_ds.SetProjection(_my_projection_)
gdal_ds.SetGeoTransform(_my_transformation_)
# Creation of the bands and scaled matrix
band = gdal_ds.GetRasterBand(1)
data = var[_chosen_index_]
data = ma.getdata(data)
data_scaled = np.interp(data, (data.min(), data.max()),
(0, 255))
data_scaled2 = data_scaled.astype(int) # This is to
rescale into integers so that it can color the layer
# *** Lines to set up the color palette ***
# Write the array to band once everything has been rescaled
band.WriteArray(data_scaled2)
gdal_ds.FlushCache()
This procedure seems to work well and takes a reasonable amount of
time. Then, the time is consumed when doing the rendering itself.
I'm not entirely sure what you mean by 'sub-optimal raster format'
or 'thre are overviews missing' so I cannot address them specifically.
Thanks,
Arturo
Martin Dobias wrote:
Hi
On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer
<qgis-developer@lists.osgeo.org> wrote:
Hello,
One of my project would greatly benefit from a much faster
rendering. After some testing, the bulk (>99%) of the time is
spent at rendering the layer with QgsRasterLayer(...,...). What
I have also noticed is that rendering time does not change with
the number of cores, which suggests that the renderer might be
using a single thread. I checked the API documentation but
couldn't find anything about multithreading or some other
procedure to accelerate rendering. I was wondering if anyone
has any suggestion or has faced a similar difficulty.
Multi-threaded rendering is done at the granularity of whole map
layers, so a single layer is rendered by a single CPU core.
Therefore if you have just one layer that is slow to render, the
multi-threading is not going to help. It is best to do some testing
how you can improve the layer's rendering - for example, it could
be that you are using some sub-optimal raster format, or there are
overviews missing, or something else... one would need more details.
Regards
Martin
_______________________________________________
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer
--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei derwww.foss-academy.com
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
johannes.kroe...@wheregroup.com
www.wheregroup.com
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
Hi Arturo,
make sure you create overviews on your big rasters or QGIS will
re-read the whole file all the time to render an appropriate
representation in the map.
Cheers, Hannes
Am 30.05.23 um 04:35 schrieb afernandez via QGIS-Developer:
Hello Martin,
I'm trying to understand what you're saying but my background is in
numerical methods where tasks are split into threads for
acceleration purposes. If a single layer is rendered by a single
core (as you write and I was observing), how would a 1,000 x 1,000
(or a 5,000 x 5,000) layer be handled? Wouldn't it take too long?
(However, you want to define 'too long').
As far as more details about the process, the tensor 'var' (n x m x
index where I have tried different combinations but let's say n x m
is 100 x 200) of floating point numbers is manipulated according to
the following pseudo-code:
# Initial manipulations
dims = var.dimensions
shape = var.shape
driver_name = 'GTIFF'
driver = gdal.GetDriverByName(driver_name)
np_dtype = var.dtype
type_code =
gdal_array.NumericTypeCodeToGDALTypeCode(np_dtype)
gdal_ds = driver.Create(_my_path_, cols, rows, 1,
gdal.GDT_UInt16)
gdal_ds.SetProjection(_my_projection_)
gdal_ds.SetGeoTransform(_my_transformation_)
# Creation of the bands and scaled matrix
band = gdal_ds.GetRasterBand(1)
data = var[_chosen_index_]
data = ma.getdata(data)
data_scaled = np.interp(data, (data.min(), data.max()),
(0, 255))
data_scaled2 = data_scaled.astype(int) # This is to
rescale into integers so that it can color the layer
# *** Lines to set up the color palette ***
# Write the array to band once everything has been rescaled
band.WriteArray(data_scaled2)
gdal_ds.FlushCache()
This procedure seems to work well and takes a reasonable amount of
time. Then, the time is consumed when doing the rendering itself.
I'm not entirely sure what you mean by 'sub-optimal raster format'
or 'thre are overviews missing' so I cannot address them specifically.
Thanks,
Arturo
Martin Dobias wrote:
Hi
On Fri, May 26, 2023 at 11:30 PM afernandez via QGIS-Developer
<qgis-developer@lists.osgeo.org> wrote:
Hello,
One of my project would greatly benefit from a much faster
rendering. After some testing, the bulk (>99%) of the time is
spent at rendering the layer with QgsRasterLayer(...,...). What
I have also noticed is that rendering time does not change with
the number of cores, which suggests that the renderer might be
using a single thread. I checked the API documentation but
couldn't find anything about multithreading or some other
procedure to accelerate rendering. I was wondering if anyone
has any suggestion or has faced a similar difficulty.
Multi-threaded rendering is done at the granularity of whole map
layers, so a single layer is rendered by a single CPU core.
Therefore if you have just one layer that is slow to render, the
multi-threading is not going to help. It is best to do some testing
how you can improve the layer's rendering - for example, it could
be that you are using some sub-optimal raster format, or there are
overviews missing, or something else... one would need more details.
Regards
Martin
_______________________________________________
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer
--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei derwww.foss-academy.com
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
johannes.kroe...@wheregroup.com
www.wheregroup.com
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
_______________________________________________
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info:https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe:https://lists.osgeo.org/mailman/listinfo/qgis-developer
--
Johannes Kröger / GIS-Entwickler/-Berater
---------------------------------------------
Aufwind durch Wissen!
Web-Seminare und Online-Schulungen
bei derwww.foss-academy.com
---------------------------------------------
WhereGroup GmbH
c/o KK03 GmbH
Lange Reihe 29
20099 Hamburg
Germany
Tel: +49 (0)228 / 90 90 38 - 36
Fax: +49 (0)228 / 90 90 38 - 11
johannes.kroe...@wheregroup.com
www.wheregroup.com
Geschäftsführer:
Olaf Knopp, Peter Stamm
Amtsgericht Bonn, HRB 9885
-------------------------------
_______________________________________________
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer