Re: [Haskell-cafe] Increasing memory use in stream computation

2013-10-14 Thread Arie Peterson
Hi Claude, On Thursday 10 October 2013 20:05:37 I wrote: Although, maybe I can do all the logic of the small function in the list monad, and stream the resulting list, as you do in the above. I tried a corresponding variant of my full program, but the memory use is quite a lot higher at the

[Haskell-cafe] Increasing memory use in stream computation

2013-10-10 Thread Arie Peterson
(Sorry for the long email.) Summary: why does the attached program have non-constant memory use? Introduction I've written a program to do a big computation. Unfortunately, the computation takes a very long time (expectedly), and the memory use increases slowly (unexpectedly),

Re: [Haskell-cafe] Increasing memory use in stream computation

2013-10-10 Thread Claude Heiland-Allen
Hi Arie, On 10/10/13 14:02, Arie Peterson wrote: (Sorry for the long email.) Summary: why does the attached program have non-constant memory use? Looking at the heap profile graph (generated with +RTS -h, no need to compile with profiling) I see the increasing memory use is split about

Re: [Haskell-cafe] Increasing memory use in stream computation

2013-10-10 Thread Bertram Felgenhauer
Arie Peterson wrote: (Sorry for the long email.) Summary: why does the attached program have non-constant memory use? Unfortunately, I don't know. I'll intersperse some remarks and propose an alternative to stream fusion at the end, which allows your test program to run in constant space.

Re: [Haskell-cafe] Increasing memory use in stream computation

2013-10-10 Thread Arie Peterson
Hi Claude, Looking at the heap profile graph (generated with +RTS -h, no need to compile with profiling) I see the increasing memory use is split about evenly between STACK and BLACKHOLE. I don't know what that means or why it occurs, but replacing `small` solved that problem for me:

Re: [Haskell-cafe] Increasing memory use in stream computation

2013-10-10 Thread Arie Peterson
Hi Bertram, Unfortunately, I don't know. I'll intersperse some remarks and propose an alternative to stream fusion at the end, which allows your test program to run in constant space. A quicker way to spot the increased memory usage is to look at GC statistics. I used ./Test +RTS