On Fri, Nov 4, 2016 at 3:51 PM, Moritz Lennert <[email protected]> wrote: > On 04/11/16 14:29, Markus Metz wrote: >> >> On Wed, Nov 2, 2016 at 5:15 PM, Markus Metz >> <[email protected]> wrote: >>> >>> On Wed, Nov 2, 2016 at 3:47 PM, Moritz Lennert >>> <[email protected]> wrote: >>>> >>>> On 02/11/16 11:05, James Duffy wrote: >>>>> >>>>> >>>>> I can't register with osgeo to submit a bug as no-one has replied with >>>>> a >>>>> 'mantra' for me to do so... what a convoluted bug reporting system! >>>>> >>>>> And in your opinion Moritz, does it look like my workflow will not be >>>>> possible unless I find a 64bit machine with a decent amount of RAM? >>>> >>>> >>>> >>>> I'm not sure about the necessity of 64bit, but more RAM probably. >>>> >>>> IIUC, all the stats of all the zones are put into memory during the run >>>> of >>>> the module. Maybe a file-based store of the information might be >>>> possible >>>> which would avoid this memory usage, but I'm not sure. >>>> >>>> @MarkusM: could you have a look at this ? See >>>> https://lists.osgeo.org/pipermail/grass-user/2016-October/075493.html >>>> for >>>> the beginning of the thread. In short: James seems to be hitting memory >>>> limits of r.univar and I'm wondering out loud whether there is something >>>> that could be done about this... >>> >>> >>> As Anna wrote, the -e flag increases memory consumption of r.univar >>> considerably. A file based approach for extended stats of r.univar is >>> difficult because the number of values per segment is initially >>> unknown to r.univar, thus the amount of disk space needed for extended >>> stats for each segment is unknown. >>> >>> I would rather use r.univar for standard stats and r.stats.quantile >>> for extended stats. >> >> >> There were some bugs in r.stats.quantile, fixed in r69770-2 in all G7 >> branches. > > > Great, thanks ! > >> >> About the limit to MAX_CATS categories in the base map (ticket #3198), >> there is not really a reason for it. I guess it is meant to protect >> against out-of-memory errors. > > > Does this mean we can just take out the lines referencing it ?
Maybe replace the error with a warning. I tested with a base map with 1.4 million categories (output of i.segment) and it works, but I needed to reduce the number of bins from 1000 to 100 to avoid OOM. With a low number of bins, the memory consumption of r.stats.quantile is much less than what r.univar requires, but a low number of bins can in some cases (histogram of the cell values with a peak) lead to higher memory consumption. r.univar does memory reallocation on demand which means that for a short time, double the amount of memory is needed. r.univar2 makes a guess about how much memory is required and then allocates all at once. Still, the binning approach of r.(stats.)quantile usually needs less memory. Markus M _______________________________________________ grass-user mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-user
