Re: [Qgis-developer] Large raster (ecw) identify very long
Guys, you are incredible! It works really well and fast. Furthermore, I don't know if you did something, but ecw drawing is instantaneous now. I have never seen that on any GIS! Thanks a lot Cheers Régis -- View this message in context: http://osgeo-org.1560.n6.nabble.com/Large-raster-ecw-identify-very-long-tp4996953p4997267.html Sent from the Quantum GIS - Developer mailing list archive at Nabble.com. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
On Wed, Aug 22, 2012 at 8:34 PM, Even Rouault even.roua...@mines-paris.org wrote: I suppose your workaround in QGIS will be to read 1x2 pixel or something like that. Yes, I have used 2x2. Hum, I humbly suggest that the fix should be done in the ECW driver rather than in QGIS. The issue with the 2x2 workaround is that if other GDAL drivers have optimizations for the 1x1 pixel use case (which would be reasonable), they will be unused now. Until it get fixed in GDAL we can use if ( GDALGetDriverShortName() == ECW) and once you fix that #if defined(GDAL_VERSION_NUM) GDAL_VERSION_NUM x.x if ( GDALGetDriverShortName() == ECW) Radim ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
On Thu, Aug 23, 2012 at 10:41 AM, haubourg regis.haubo...@eau-adour-garonne.fr wrote: Guys, you are incredible! It works really well and fast. Furthermore, I don't know if you did something, but ecw drawing is instantaneous now. I have never seen that on any GIS! On Wed, Aug 22, 2012 at 11:40 AM, haubourg regis.haubo...@eau-adour-garonne.fr wrote: Hi all, I'm ready to support developpement now ... You can find detailed instructions here: http://www.qgis.org/en/sponsorship.html Radim ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
Until it get fixed in GDAL we can use if ( GDALGetDriverShortName() == ECW) and once you fix that #if defined(GDAL_VERSION_NUM) GDAL_VERSION_NUM x.x if ( GDALGetDriverShortName() == ECW) Seems reasonnable. Except I suggest using a string comparison function and not == ;-) Would you mind creating a ticket in GDAL trac about that ? Radim ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
On Wed, Aug 22, 2012 at 11:40 AM, haubourg regis.haubo...@eau-adour-garonne.fr wrote: Hi all, I'm ready to support developpement now, so let's undig a topic discussed previously here: http://hub.qgis.org/issues/4594 http://hub.qgis.org/issues/4594 Identify for very large ecw (from 1 to 100 Go - 1 080 000 * 1 090 000 pixels) is extremely long (more than 1min) and results to memory leak. Very often, this only gives a useless RGB info because ecw are orthophotos or topographic scans. This is a problem if identify mode queries all layers under mouse clic. 1 - Memory leak can be solved by disabling Gdal cache for ecw says Nathan.* Do you confirm ? Can you give more details so that I write a contract for it? * QGIS is using GDALRasterIO() which reads a single pixel on original resolution. AFAIK, ECW is using tiles internally so it should be all very fast. I can imagine 2 problems: - the tiles in ECW file are too big - can you verify somehow how big are the tiles? - GDAL is reading bigger portion of data than necessary (cc to GDAL list) Is it drawing of the raster also so slow (on raster resolution zoom)? Radim 2 - *I would also like to add a behaviour to handle very long response time, I would really like to have your feedback on it:* - Option 1 : Disable identify for very large raster layers in project properties when raster is loaded. Keep QGS properties if raster is loaded when opening a project file. Should we add options to configure that in general options ( Disable very large raster identify + raster size threshold) - Option 2 : Offer a checkbox in identify mode options that could be named disable raster identify - Option 3 : Considering it's a very particular problem for huge ecw users and let's make a plugin that deactivates identify for big raster. I must say I don't vote for here because most french public user I know are using big ecw, and I think it might be harder and more expensive to do this from python.. (and adds a python plugin to load, adds a thing more to handle on enterprise config.. ) *I vote for Option 1* Thoughts guys? (And welcome to Victor by the way, congratulations!) -- View this message in context: http://osgeo-org.1560.n6.nabble.com/Large-raster-ecw-identify-very-long-tp4996953.html Sent from the Quantum GIS - Developer mailing list archive at Nabble.com. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
Radim Blazek-2 wrote QGIS is using GDALRasterIO() which reads a single pixel on original resolution. AFAIK, ECW is using tiles internally so it should be all very fast. I can imagine 2 problems: - the tiles in ECW file are too big - can you verify somehow how big are the tiles? Hi Radim, I'm not ecw specialist, but I understand it is a wavelet compression algorithm, not a tiled one. There is one big image with pyramids inside. Radim Blazek-2 wrote - GDAL is reading bigger portion of data than necessary (cc to GDAL list) Is it drawing of the raster also so slow (on raster resolution zoom)? Radim No everything is fine when drawing data, whatever resolution I use, this is the big thing with ecw. I must say I find that it was a bit faster with 1.7.4 than 1.8 and later. But this could be related to other factors like internal network capacities.. I haven't benched on it yet. -- View this message in context: http://osgeo-org.1560.n6.nabble.com/Large-raster-ecw-identify-very-long-tp4996953p4996972.html Sent from the Quantum GIS - Developer mailing list archive at Nabble.com. ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
On Wed, Aug 22, 2012 at 12:29 PM, haubourg
regis.haubo...@eau-adour-garonne.fr wrote:
Radim Blazek-2 wrote
QGIS is using GDALRasterIO() which reads a single pixel on original
resolution. AFAIK, ECW is using tiles internally so it should be all
very fast. I can imagine 2 problems:
- the tiles in ECW file are too big - can you verify somehow how big
are the tiles?
Hi Radim, I'm not ecw specialist, but I understand it is a wavelet
compression algorithm, not a tiled one.
There is one big image with pyramids inside.
I am not sure, but I thought that it is using wavelet, pyramids and
tiles. Without tiles it could not be fast enough on higher resolution.
Radim Blazek-2 wrote
- GDAL is reading bigger portion of data than necessary (cc to GDAL list)
Is it drawing of the raster also so slow (on raster resolution zoom)?
Radim
No everything is fine when drawing data, whatever resolution I use, this is
the big thing with ecw.
I must say I find that it was a bit faster with 1.7.4 than 1.8 and later.
But this could be related to other factors like internal network
capacities.. I haven't benched on it yet.
I found in GDAL ecwdataset.cpp that it is treating single row
requests in IRasterIO in a special way:
if( nYSize == 1 )
{
return GDALRasterBand::IRasterIO(eRWFlag, nXOff, nYOff, nXSize, nYSize,
pData, nBufXSize, nBufYSize,
eBufType, nPixelSpace, nLineSpace );
}
That could be the difference between draw and identify. Are you able
to test with current master if I send you a patch for QGIS GDAL
provider?
Radim
--
View this message in context:
http://osgeo-org.1560.n6.nabble.com/Large-raster-ecw-identify-very-long-tp4996953p4996972.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
Selon Radim Blazek radim.bla...@gmail.com:
On Wed, Aug 22, 2012 at 12:29 PM, haubourg
regis.haubo...@eau-adour-garonne.fr wrote:
Radim Blazek-2 wrote
QGIS is using GDALRasterIO() which reads a single pixel on original
resolution. AFAIK, ECW is using tiles internally so it should be all
very fast. I can imagine 2 problems:
- the tiles in ECW file are too big - can you verify somehow how big
are the tiles?
Hi Radim, I'm not ecw specialist, but I understand it is a wavelet
compression algorithm, not a tiled one.
There is one big image with pyramids inside.
I am not sure, but I thought that it is using wavelet, pyramids and
tiles. Without tiles it could not be fast enough on higher resolution.
Radim Blazek-2 wrote
- GDAL is reading bigger portion of data than necessary (cc to GDAL list)
Is it drawing of the raster also so slow (on raster resolution zoom)?
Radim
No everything is fine when drawing data, whatever resolution I use, this is
the big thing with ecw.
I must say I find that it was a bit faster with 1.7.4 than 1.8 and later.
But this could be related to other factors like internal network
capacities.. I haven't benched on it yet.
I found in GDAL ecwdataset.cpp that it is treating single row
requests in IRasterIO in a special way:
if( nYSize == 1 )
{
return GDALRasterBand::IRasterIO(eRWFlag, nXOff, nYOff, nXSize,
nYSize,
pData, nBufXSize, nBufYSize,
eBufType, nPixelSpace, nLineSpace );
}
Regis,
what are the dimensions (in pixels) of your big ECW ?
Radim,
I tried the following Python script that must be representative of how QGIS must
do picking (I guess it does a RasterIO(, x, y, 1, 1, ... 1, 1) )
from osgeo import gdal
import random
import sys
ds = gdal.Open(sys.argv[1])
xsize = ds.RasterXSize
ysize = ds.RasterYSize
while True:
x = random.randint(0, xsize)
y = random.randint(0, ysize)
data = ds.ReadRaster(x,y,1,1)
After some time, the python process occupies the 1/4 of the total RAM. This is
the default behaviour of the ECW SDK documented in
http://gdal.org/frmt_ecw.html. I then set ECW_CACHE_MAXMEM=100 (1 million
bytes) and the memory usage was very small as expected. So I believe there is no
memory leak in the driver. Note: in the old ECW SDK 3.3 (I don't know for the
newer ones), there was a bug in some cases : if the RAM was 8 GB, RAM / 4 2
GB which overflowed a 32 bit, resulting in the SDK allocating memory without
limit.
The ECW driver is quite complex in its reading strategy to establish views,
but from what I've captured and you noticed, when you ask a window with a 1
pixel height, it goes to IReadBlock() which will fetch one entire line and put
it in the GDAL cache. This is to avoid issuing a SetView() each time a line is
read. The intent here is to be clever for the common line-by-line pattern
access. The drawback of this is that if the raster has a big width and you only
want to read one single pixel, the cost for reading the whole line might be much
greater than the cost of establishing SetView() for your single pixel.
I suppose your workaround in QGIS will be to read 1x2 pixel or something like
that.
Ultimately, we would probably need the following GDAL patch. I can't really say
if it will improve performance a lot because my biggest ECW is just
1x1.
Index: ecwdataset.cpp
===
--- ecwdataset.cpp (revision 24824)
+++ ecwdataset.cpp (working copy)
@@ -246,8 +246,10 @@
/* We will drop down to the block oriented API if only a single*/
/* scanline was requested. This is based on the assumption that*/
/* doing lots of single scanline windows is expensive. */
+/* But for single pixel reading (picking use case), this is not a */
+/* good strategy for big rasters. */
/* */
-if( nYSize == 1 )
+if( nYSize == 1 nXSize != 1 )
{
#ifdef NOISY_DEBUG
CPLDebug( ECWRasterBand,
@@ -1038,7 +1040,7 @@
/* If we are requesting a single line at 1:1, we do a multi-band */
/* AdviseRead() and then TryWinRasterIO() again. */
/* */
-if( nYSize == 1 nBufYSize == 1 nBandCount 1 )
+if( nYSize == 1 nXSize != 1 nBufYSize == 1 nBandCount 1 )
{
CPLErr eErr;
@@ -1064,8 +1066,8 @@
/* band case where we post a big window for the view, and allow*/
/* sequential reads. */
/* */
-if( nXSize nBufXSize || nYSize nBufYSize || nYSize == 1
-|| nBandCount 100 || nBandCount == 1 || nBufYSize == 1
+
Re: [Qgis-developer] Large raster (ecw) identify very long
Even, thanks for exhaustive explanation and testing. On Wed, Aug 22, 2012 at 2:37 PM, Even Rouault even.roua...@mines-paris.org wrote: I found in GDAL ecwdataset.cpp that it is treating single row requests in IRasterIO in a special way: I tried the following Python script that must be representative of how QGIS must do picking (I guess it does a RasterIO(, x, y, 1, 1, ... 1, 1) ) Yes. I suppose your workaround in QGIS will be to read 1x2 pixel or something like that. Yes, I have used 2x2. Radim ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
On Wed, Aug 22, 2012 at 3:25 PM, haubourg regis.haubourg@eau-adour- Radim, I'll try to to check if tiles are present, but I don't think so. We compressed it with ERmapper without tile option. No more important, we know already the problem. I'm afraid I'm not yet able to test patches since I never compiled QGIS and don't really have machines, skills and time to do that. I really have to set up a dev machine. I could eventually send you a big ecw ? (10 Go on ftp) It should be fixed in master a795a5d334. Please test tomorrow with Win night build, I am not sure however if it includes ECW support. Do I understand that it might be possible to solve directly the problem, without having to add workaround options on project propertis as I suggested? Yes. Radim ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [Qgis-developer] Large raster (ecw) identify very long
Le mercredi 22 août 2012 20:13:39, Radim Blazek a écrit : Even, thanks for exhaustive explanation and testing. On Wed, Aug 22, 2012 at 2:37 PM, Even Rouault even.roua...@mines-paris.org wrote: I found in GDAL ecwdataset.cpp that it is treating single row requests in IRasterIO in a special way: I tried the following Python script that must be representative of how QGIS must do picking (I guess it does a RasterIO(, x, y, 1, 1, ... 1, 1) ) Yes. I suppose your workaround in QGIS will be to read 1x2 pixel or something like that. Yes, I have used 2x2. Hum, I humbly suggest that the fix should be done in the ECW driver rather than in QGIS. The issue with the 2x2 workaround is that if other GDAL drivers have optimizations for the 1x1 pixel use case (which would be reasonable), they will be unused now. Radim ___ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer
