Hi James,
On 04/11/16 22:08, James Duffy wrote:
Sorry to post again on this long thread...
No worries :-)
I have just managed to get my hands on a 64bit osgeolive install with
12GB RAM available and the afformentioned function worked. Just to
reiterate this was i.segment.stats on gp_seg_optimum_clump (which was
the output of i.segment, run through r.clump). I haven't created the
vectormap, but it seems that the csv was created correctly.
I don't know if it was the 64bit or the bit of extra RAM that made the
difference here.
I would guess the extra RAM, but am no expert on these questions.
I'm working on a revised version of i.segment.stats to use
r.stats.quantile. It would be great if you could test that once its done.
Moritz
James
On 4 November 2016 at 15:34, Markus Metz <[email protected]
<mailto:[email protected]>> wrote:
On Fri, Nov 4, 2016 at 3:51 PM, Moritz Lennert
<[email protected] <mailto:[email protected]>>
wrote:
> On 04/11/16 14:29, Markus Metz wrote:
>>
>> On Wed, Nov 2, 2016 at 5:15 PM, Markus Metz
>> <[email protected]
<mailto:[email protected]>> wrote:
>>>
>>> On Wed, Nov 2, 2016 at 3:47 PM, Moritz Lennert
>>> <[email protected]
<mailto:[email protected]>> wrote:
>>>>
>>>> On 02/11/16 11:05, James Duffy wrote:
>>>>>
>>>>>
>>>>> I can't register with osgeo to submit a bug as no-one has
replied with
>>>>> a
>>>>> 'mantra' for me to do so... what a convoluted bug reporting
system!
>>>>>
>>>>> And in your opinion Moritz, does it look like my workflow will
not be
>>>>> possible unless I find a 64bit machine with a decent amount of
RAM?
>>>>
>>>>
>>>>
>>>> I'm not sure about the necessity of 64bit, but more RAM probably.
>>>>
>>>> IIUC, all the stats of all the zones are put into memory during
the run
>>>> of
>>>> the module. Maybe a file-based store of the information might be
>>>> possible
>>>> which would avoid this memory usage, but I'm not sure.
>>>>
>>>> @MarkusM: could you have a look at this ? See
>>>>
https://lists.osgeo.org/pipermail/grass-user/2016-October/075493.html
<https://lists.osgeo.org/pipermail/grass-user/2016-October/075493.html>
>>>> for
>>>> the beginning of the thread. In short: James seems to be
hitting memory
>>>> limits of r.univar and I'm wondering out loud whether there is
something
>>>> that could be done about this...
>>>
>>>
>>> As Anna wrote, the -e flag increases memory consumption of r.univar
>>> considerably. A file based approach for extended stats of
r.univar is
>>> difficult because the number of values per segment is initially
>>> unknown to r.univar, thus the amount of disk space needed for
extended
>>> stats for each segment is unknown.
>>>
>>> I would rather use r.univar for standard stats and r.stats.quantile
>>> for extended stats.
>>
>>
>> There were some bugs in r.stats.quantile, fixed in r69770-2 in all G7
>> branches.
>
>
> Great, thanks !
>
>>
>> About the limit to MAX_CATS categories in the base map (ticket
#3198),
>> there is not really a reason for it. I guess it is meant to protect
>> against out-of-memory errors.
>
>
> Does this mean we can just take out the lines referencing it ?
Maybe replace the error with a warning. I tested with a base map with
1.4 million categories (output of i.segment) and it works, but I
needed to reduce the number of bins from 1000 to 100 to avoid OOM.
With a low number of bins, the memory consumption of r.stats.quantile
is much less than what r.univar requires, but a low number of bins can
in some cases (histogram of the cell values with a peak) lead to
higher memory consumption.
r.univar does memory reallocation on demand which means that for a
short time, double the amount of memory is needed. r.univar2 makes a
guess about how much memory is required and then allocates all at
once. Still, the binning approach of r.(stats.)quantile usually needs
less memory.
Markus M
_______________________________________________
grass-user mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/grass-user
