Even, Thanks for checking into this and the ideas.
Eli On Tue, Apr 16, 2013 at 10:18 AM, Even Rouault <[email protected]> wrote: > Le lundi 15 avril 2013 23:47:33, Eli Adam a écrit : >> Hi all, >> >> I'm adding internal overviews to a ~20G tiff file and I get this error >> (1000+ times): >> gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config >> COMPRESS_OVERVIEW PACKBITS >> ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields >> array (1410046 elements of 16 bytes each) >> gdaladdo still completes and the file seems to work fine. What does >> that error mean? Does it impact the output file or overviews? Is >> there a different way I should approach this? Are there any general >> rules for which compression to select? I've found JPEG-In-TIFF, using >> PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure >> of the best compression for the type of raster (in this case single >> band, byte, colormap). > > That's interesting. I've tried to reproduce your setup with an artificial > blank > input file of the same dimensions and here's what I get (after several hours): > > $ gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config > COMPRESS_OVERVIEW PACKBITS > 0...10...20...30...40...50...60...70...80...90...100 - done. > Current VSI memory usage : 0 bytes > Maximum VSI memory usage : 2361761086 bytes > Number of calls to VSIMalloc() : 43981823 > Number of calls to VSICalloc() : 1259 > Number of calls to VSIRealloc() : 17378556 > Number of calls to VSIFree() : 43983082 > VSIMalloc + VSICalloc - VSIFree : 0 > > The interesting part is the memory usage report. It is based on an > instrumented version of GDAL that instruments each memory allocation routine. > This is with the internal version of libtiff that redirects the TIFF memory > allocation routines to the GDAL ones. So apparently there wouldn't be a memory > leak (some valgrind'ing should be done on a smaller file to confirm), but a > maximum memory usage of 2.3 GB ! No wonder why it fails on a Windows 32 bit > (my test was done a Linux 64bit). I'm not sure why such a high memory use > occurs. Some investigation should be done. > > As far as the error message you've seen, I wouldn't swear that there's not > some pixel corruption in the result you get... > > As a potential track for a workaround (completely untested), perhaps you could > try to do : > > $ gdaladdo topo_mosaic_4326.tif 2 --config COMPRESS_OVERVIEW PACKBITS > $ gdaladdo topo_mosaic_4326.tif 4 --config COMPRESS_OVERVIEW PACKBITS > $ gdaladdo topo_mosaic_4326.tif 8 --config COMPRESS_OVERVIEW PACKBITS > $ gdaladdo topo_mosaic_4326.tif 16 --config COMPRESS_OVERVIEW PACKBITS > etc.. > > > _______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
