Re: [gdal-dev] gdalwarp: ERROR 2: Out of memory allocating {number} byte destination buffer.

2011-06-08 Thread Matt Wilkie 

Hi Frank,

Thanks for the clear explanation. I've added, with light editing, to 
http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp.



I imagine you will just have to use more modest buffers or else get a 64bit
executable for gdalwarp.


Running gdalwarp with defaults (no options specified) seems to be 
working fine (at input #24 and no errors).


Looks like always gunning for performance can be problematic. ;-)

-matt

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] gdalwarp: ERROR 2: Out of memory allocating {number} byte destination buffer.

2011-06-08 Thread Frank Warmerdam 

On 11-06-08 03:05 PM, Matt Wilkie wrote:

Hi Folks,

Can anyone tell me what this error means and how to work around it?

Win7 x64 command shell
osgeo4w gdal 1.8
13gb ram (~4.7gb free according to Process Explorer)
121gb free on the drive (mapped to network)
20gb free on drive pointed to by %TEMP%

There are 26 input files whose size ranges from 110mb to 260mb, totalling
5.5gb(ftp://ftp.geomaticsyukon.ca/DEMs/CDED_50k/)

If I up GDAL_CACHEMAX and -wm to higher value, like 2048, I get the error on
every single input file instead of just some (I'm still waiting for the results
of not using those options at all).

-
*gdalinfo --version
*GDAL 1.8.0, released 2011/01/12*

set .gdopt=*--config GDAL_CACHEMAX 500 -wm 500 --config bigtiff if_safer -multi

...


Processing input file ..\dem_115F.tif.
Using internal nodata values (eg. -3.40282e+038) for image ..\dem_115F.tif.
0...10...20...30...40...50ERROR 2: Out of memory allocating 365425784 byte
destination buffer.


Matt,

Despite being on Win7 x64 OSGeo4W is still all 32bit executables and
they operate in a 32bit process space.  My understanding is that Win32
processes are frequently subject to memory fragmentation problems and
so even in a process with - in theory - 2GB of heap RAM space available
it is still often difficult to allocate large blocks of memory.   In this
case with -wm 500 set the main warp buffers can be quite large (364MB
for one in your case) and it seems it has failed to allocate the buffer.

I imagine you will just have to use more modest buffers or else get a 64bit
executable for gdalwarp.

Best regards,
--
---+--
I set the clouds in motion - turn up   | Frank Warmerdam, warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Programmer for Rent

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] gdalwarp: ERROR 2: Out of memory allocating {number} byte destination buffer.

2011-06-08 Thread Benjamin Welton 

Hey Matt,

If i remember correctly osgeo4w is 32bit (its been a while since iv 
looked at the osgeo4w binaries however so i may be incorrect). If they 
are 32bit and you are setting -wm to 2048 (2GB) you might be having 
memory limitation issues (which i believe 2GB virtual memory for x86 
executables on windows).


Ben



On 06/08/2011 02:05 PM, Matt Wilkie wrote:

Hi Folks,

Can anyone tell me what this error means and how to work around it?

Win7 x64 command shell
osgeo4w gdal 1.8
13gb ram (~4.7gb free according to Process Explorer)
121gb free on the drive (mapped to network)
20gb free on drive pointed to by %TEMP%

There are 26 input files whose size ranges from 110mb to 260mb, 
totalling 5.5gb(ftp://ftp.geomaticsyukon.ca/DEMs/CDED_50k/)


If I up GDAL_CACHEMAX and -wm to higher value, like 2048, I get the 
error on every single input file instead of just some (I'm still 
waiting for the results of not using those options at all).


-
*gdalinfo --version
*GDAL 1.8.0, released 2011/01/12*

set .gdopt=*--config GDAL_CACHEMAX 500 -wm 500 --config bigtiff 
if_safer -multi
*set .input=*..\dem_105E.tif ..\dem_105L.tif ..\dem_105M.tif 
..\dem_105N.tif ..\dem_106C.tif ..\dem_106D.tif ..\dem_106E.tif 
..\dem_106L.tif ..\dem_115F.tif ..\dem_115G.tif ..\dem_115H.tif 
..\dem_115I.tif ..\dem_115J.tif ..\dem_115K.tif ..\dem_115N.tif 
..\dem_115O.tif ..\dem_115P.tif ..\dem_116A.tif ..\dem_116B.tif 
..\dem_116C.tif ..\dem_116F.tif ..\dem_116G.tif ..\dem_116H.tif 
..\dem_116I.tif ..\dem_116J.tif ..\dem_116K.tif


*gdalwarp  %.gdopt% -t_srs EPSG:26907 -r bilinear -tr 30 30 %.input% 
dem.tif*

Creating output file that is 15662P x 23332L.
Processing input file ..\dem_105E.tif.
Using internal nodata values (eg. -3.40282e+038) for image 
..\dem_105E.tif.

0...10...20...30...40...50...60...70...80...90...100 - done./
/

   /next 7 inputs processed successfully/

Processing input file ..\dem_115F.tif.
Using internal nodata values (eg. -3.40282e+038) for image 
..\dem_115F.tif.
0...10...20...30...40...50ERROR 2: Out of memory allocating 365425784 
byte destination buffer.

...60...70..Processing input file ..\dem_115G.tif.
Using internal nodata values (eg. -3.40282e+038) for image 
..\dem_115G.tif.

.80...90...100 - done.
Processing input file ..\dem_115H.tif.

   /Error at 50% on dem_115f.tif, but carries on /
   /then starts a new input, dem_115g.tif, at 70%+/
   /and apparently completes 115f, or is that 115g?/
   /in any case the next tile, dem_115h, is successful: /

Processing input file ..\dem_115H.tif.
Using internal nodata values (eg. -3.40282e+038) for image 
..\dem_115H.tif.

0...10...20...30...40...50...60...70...80...90...100 - done.

   /next 2 are successful,
   followed by 13 errors./

Processing input file ..\dem_116J.tif.
Using internal nodata values (eg. -3.40282e+038) for image 
..\dem_116J.tif.

ERROR 2: Out of memory allocating 365425784 byte destination buffer.
Processing input file ..\dem_116K.tif.
Using internal nodata values (eg. -3.40282e+038) for image 
..\dem_116K.tif.

ERROR 2: Out of memory allocating 365425784 byte destination buffer.







___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev