That seemed to do the trick. It finished overnight, a lot faster than before.
Final file was 83GB. (Only 1/2 the dataset was used) Is there a reason GDAL_CACHEMAX 2147 is the maximum? Under 64-bit a 32-bit process has 4GB of virtual memory rather than 2GB under 32-bit (or 3GB with the /3GB Windows boot switch). I was successfully able to use kdu_compress on the result as well so now I only need 4GB to store the file! (Although kdu_compress also has a memory limit at 2GB, but you can flush the codestream after every tile to fix that). Thanks, Leith Bade [email protected] On 17 July 2010 15:29, Greg Coats <[email protected]> wrote: > Since you have 4 GB RAM, I suggest overriding the 40 MB GDAL_CACHEMAX > default, and increasing it > gdal_merge.py --config GDAL_CACHEMAX 1000 > The maximum possible GDAL_CACHEMAX is > gdal_merge.py --config GDAL_CACHEMAX 2147 > Greg > > On Jul 16, 2010, at 9:19 PM, Leith Bade wrote: > > Hi, > > I am trying to use gdal_merge to mosaic a very large topo GeoTIFF set. > > Uncompressed the data set is 60GB, but I keep it stored with DEFLATE > compression which results in a dataset under 10GB. > > Mosaicked the uncompressed file will be 125GB because of the large regions > of nodata generated. Unfortunately this is too big to store on my HDD so I > need to apply DEFLATE to it as well. > > I am experimenting with only 1/2 the data set at the moment with this > command: > gdal_merge -co COMPRESS=DEFLATE -co ZLEVEL=9 -co BIGTIFF=YES -o NI-50.tif > *-00.tif > > On my AMD Phenom II X4 @ 3.2GHz, 64 bit Windows 7, 4GB DDR3 CAS-7 RAM > (basically a decently specced PC gaming machine) > it has been running for 16 hours and so far has only made it 60% (124 of > 204 files) > > The other issue is so far the file is 54GB so I will likely run out of disc > space before it finished. This indicates that no DEFLATE compression is > happening at all! > > It currently maxing only 1 CPU core, so I assume it is trying to run > DEFLATE then somehow failing to compress at all? > > Looking at gdal_merge.py I think the major performance issue is the order > in which it copies the data. Currently it copies an entire image at a time > (1 colour channel at a time). Thus DEFLATE will not work due to the rather > random write pattern to each scanline. > > I think a much faster method would be to calculate the destination > rectangles for each file into some sort of list. Aftter this generate the > destination file 1 scanline at a time, calculating which source images > intersect, and working left to right filling with solid color or copying > scanlines from the source image. > > This allows the DEFLATE to work far more effciently as the write pattern is > horizontally linear. > > What are your views/suggestions/etc.? > > Thanks, > Leith Bade > [email protected] > > _______________________________________________ > gdal-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > > > _______________________________________________ > gdal-dev mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/gdal-dev >
_______________________________________________ gdal-dev mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/gdal-dev
