I am using GDALGetRandomRasterSample() to determine scale values for 16-bit per band map sources that I'm processing. What I've found is that for my NITF files that use JPEG2000 subimages, GDALGetRandomRasterSample() takes a long time to run and consumes a lot of memory (~800 MB RAM for a 540 MB NITF). The memory is not released until the source is closed (with GDALClose()).
One of my sources causes GDAL or the source driver (NITF? JP2ECW? JP2MrSID?) to commit an access violation by attempting to read or write addresses 0x0 or 0x4, for instance. This is a larger map (9132x37889 pixels, a 1.2 gigabyte file). Watching in the debugger, I can that GDALGetRandomRasterSample() takes about 90 seconds or so on the first raster band. At this point it is using over 900 MB RAM. Maybe 10 seconds in to processing the second raster band, it crashes with the access violation (at around a gig of memory usage, maybe less). The problem can be reproduced by running gdalinfo on the source, with the -sample option. The -stats option also fails, but only once it reaches/exceeds 2 gigs memory usage. Here is what I've attempted so far: * FWTools 2.4.7 (1.7.0b2) and 2.2.8 (1.6.0?), on Windows XP x64 (8 gigs RAM) and in a 32-bit virtual machine (2 gigs). My target environment is 32-bit XP. * I set GDAL_CACHEMAX to 900 MB, but have also tried 100 MB. * I tried setting GDAL_ECW_CACHE_MAXMEM to 200 MB. This appears to make no difference whatsoever, so I'm not positive that I'm using the JP2ECW driver. "gdalinfo --formats" shows both JP2ECW and JP2MrSID. * I've tried opening the large NITF in OpenEV under both 2.4.7 and 2.2.8. It takes a while, prompts me to select which subdataset (there are 2), I select the subdataset and then it processes for a long time and eventually crashes. Basically, it's the same behavior I see from my own code. * I also tried building GDAL myself from the 1.7.1 sources, but I cannot build the ECW driver because I do not have the required third party products. Perhaps because of this, I also could not figure out how to step into the GDAL sources for the driver from my VS2008 SP1 debugger. * I tried copying the GDALGetRandomRasterSample() into my code and stepped through it. The large memory allocations occur when GetLockedBlockRef() is called. For these NITF/JPEG2000 files, the memory is never released by DropLock(). * The memory appears to be released when the source dataset is destructed with GDALClose(). This might be a workaround for me, i.e. process one band at a time, closing and reopening the file between bands, but I am still concerned about my target platform running out of RAM, and the long time that this takes. * I'm also considering making a temporary, much smaller GeoTIFF from the source and then computing the scale values from the smaller file. I haven't tried this, yet, and would prefer an approach that runs quickly with low memory. Is this a known problem for NITF or JPEG2000 files? Is there a way to see which JPEG2000 driver is used and to switch it to the other one? Any other suggestions? Thanks, Jonathan _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
