On Wed, Aug 24, 2011 at 3:49 PM, Micha Silver <[email protected]> wrote: > On Wed, 2011-08-24 at 13:58 +0200, [email protected] wrote: >> Hello, >> >> I am trying to use the r.watershed function to calculate streams (rivers) >> from elevation data. As long as I make sure that the GRASS region is not >> too big things work fine, but in the end I would like to do the >> calculations at the resolution of the elevation data. >> >> Let me first clarify what I mean with big in this case. I can do the >> calculations with regions up to 10000x10000 pixels, but above that I start >> to run into trouble. My elevation data is about 80000x60000, so I guess >> that is quite big. >> > > There are some nice guidelines on the r.watershed man page regarding > memory requirements. Specifically: > > ------------------------------------------------------ > In-memory mode and disk swap mode > There are two versions of this program: ram and seg. ram is used by > default, seg can be used by setting the -m flag. > The ram version requires a maximum of 31 MB of RAM for 1 million cells. > Together with the amount of system memory (RAM) available, this value > can be used to estimate whether the current region can be processed with > the ram version. > The ram version uses virtual memory managed by the operating system to > store all the data structures and is faster than the seg version; seg > uses the GRASS segmentation library which manages data in disk files. > seg uses only as much system memory (RAM) as specified with the memory > option, allowing other processes to operate on the same system, even > when the current geographic region is huge. > Due to memory requirements of both programs, it is quite easy to run out > of memory when working with huge map regions. If the ram version runs > out of memory and the resolution size of the current geographic region > cannot be increased, either more memory needs to be added to the > computer, or the swap space size needs to be increased. If seg runs out > of memory, additional disk space needs to be freed up for the program to > run. The r.terraflow module was specifically designed with huge regions > in mind and may be useful here as an alternative. > --------------------------------------------------- > > So with your 80,000 X 60,000 = 4,800,000,000 cells region you would need > about 150 GB (!) of RAM to run it in memory.
This large number of cells is not yet supported in r.watershed (see similar problem with r.terraflow [0]). The maximum number of cells is 2^31 - 1 = 2,147,483,647. This limitation is 1) independent of whether the in-memory or on-disk option is used, 2) for the (recommended) on-disk option mainly imposed by the segment library of grass and applies therefore also to e.g r.cost and r.walk. > So you'll most likely need to use the -m option which writes everything > out to disk, making the process much slower. > > > >> So my first question would be: Is the file I try to process too big or has >> r.watershed been run successfully on such big files? >> >> >From the mailing list I understand GRASS needs to be compiled with large >> file support to work with such big files. I am not sure if my version is, I >> have tried in on Windows with the stable version 6.4 from the OSGeo4W >> installer and on Ubuntu with the 6.4 version from their repository. >> >> This is the error I got on Ubuntu while running r.watershed: >> >> SECTION 3 beginning: Initiating Variables. 5 sections total. >> WARNING: No such file or directory >> WARNING: cseg_open(): could not write segment file >> Floating point exception >> WARNING: Subprocess failed with exit code 34817 >> > > Not sure about these errors, THey do *not* indicate memory problems. > I guess so too. The first message indicates that the on-disk option (-m flag) is used. One reason could be not enough free disk space to create temporary files. BTW, there seems to be a cut'n paste error, the first message should read "SECTION 1 beginning: ..." and not "SECTION 3 beginning: ..." Just out of curiosity: how did you create this massive 4.8 billion cells DEM? Markus M [0] http://lists.osgeo.org/pipermail/grass-dev/2011-August/055314.html >> Any advice on how to prevent this error? >> >> Thanks, >> >> Arno >> _______________________________________________ >> grass-user mailing list >> [email protected] >> http://lists.osgeo.org/mailman/listinfo/grass-user >> >> This mail was received via Mail-SeCure System. >> >> > > > _______________________________________________ > grass-user mailing list > [email protected] > http://lists.osgeo.org/mailman/listinfo/grass-user > _______________________________________________ grass-user mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-user
