Re: [Haskell-cafe] why does the binary library require so much memory?

2009-08-16 Thread Alex Mason
Hi Don, I was wondering if perhaps this might be a slightly better instance for Binary [a], that might solve a) the problem of having to traverse the entire list first, and b) the list length limitation of using length and Ints. My version is hopefully a little more lazy (taking maxBound

[Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Jeremy Shaw
Hello, Using encode/decode from Binary seems to permamently increase my memory consumption by 60x fold. I am wonder if I am doing something wrong, or if this is an issue with Binary. If I run the following program, it uses sensible amounts of memory (1MB) (note that the bin and list' thunks

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Don Stewart
I just need a small test case to reproduce the problem. Thanks! -- Don jeremy: Hello, Using encode/decode from Binary seems to permamently increase my memory consumption by 60x fold. I am wonder if I am doing something wrong, or if this is an issue with Binary. If I run the following

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Bryan O'Sullivan
On Fri, Jul 31, 2009 at 1:56 PM, Jeremy Shaw jer...@n-heptane.com wrote: Using encode/decode from Binary seems to permamently increase my memory consumption by 60x fold. I am wonder if I am doing something wrong, or if this is an issue with Binary. It's an issue with the Binary instance for

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Don Stewart
bos: On Fri, Jul 31, 2009 at 1:56 PM, Jeremy Shaw jer...@n-heptane.com wrote: Using encode/decode from Binary seems to permamently increase my memory consumption by 60x fold. I am wonder if I am doing something wrong, or if this is an issue with Binary. It's an issue with

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Jeremy Shaw
Hello, Is there a work-around? This is killer for Happstack. Most Happstack applications use IxSet, which in turn uses lists to serialize the data to/from disk. Also, why doesn't the stuff get freed eventually? - jeremy At Fri, 31 Jul 2009 14:27:30 -0700, Don Stewart wrote: bos: On Fri,

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Don Stewart
Why don't you use your own instance to serialize IxSet lazily (or however you would like?) There's no reason to be constrained to use the [a] instance. -- Don jeremy: Hello, Is there a work-around? This is killer for Happstack. Most Happstack applications use IxSet, which in turn uses

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Jason Dusek
2009/07/31 Jeremy Shaw jer...@n-heptane.com: ...why doesn't the stuff get freed eventually? It is my understanding that the GHC runtime never lets go of memory once it has requested it. (Confirmation either way would be informative.) -- Jason Dusek

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Don Stewart
jason.dusek: 2009/07/31 Jeremy Shaw jer...@n-heptane.com: ...why doesn't the stuff get freed eventually? It is my understanding that the GHC runtime never lets go of memory once it has requested it. (Confirmation either way would be informative.) It doesn't return memory to the OS.

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Jeremy Shaw
At Fri, 31 Jul 2009 14:49:03 -0700, Don Stewart wrote: Why don't you use your own instance to serialize IxSet lazily (or however you would like?) There's no reason to be constrained to use the [a] instance. Well, the Set instance might actually be a better choice. But the Set instance in

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Don Stewart
jeremy: At Fri, 31 Jul 2009 14:49:03 -0700, Don Stewart wrote: Why don't you use your own instance to serialize IxSet lazily (or however you would like?) There's no reason to be constrained to use the [a] instance. Well, the Set instance might actually be a better choice. But the

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Jeremy Shaw
At Fri, 31 Jul 2009 15:43:49 -0700, Don Stewart wrote: Oh, it is entirely possible to use a different instance, that has different semantics for lists. You want to write the list incrementally? I don't think so. In happstack, the idea is to have all your state in RAM. But, since your

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Jeremy Shaw
Hrm, I think actually, that my test program was a bit bogus... investigating now. - jeremy ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Jeremy Shaw
Ok, I fixed my test program, and now things seem more reasonable. The original version was allowing the GC to collect the list in the first case, but not the second. However, I don't want the list to be collected. The new version seems to fix that issue. Now the control uses 40MB and when I

Re[2]: [Haskell-cafe] why does the binary library require so much memory?

2009-07-31 Thread Bulat Ziganshin
Hello Jeremy, Saturday, August 1, 2009, 3:15:02 AM, you wrote: So, the desired experience would be: 1. A program starts running and populates an IxSet. At this point in time n MB of RAM are being used. 2. We use Binary to snapshot the entire IxSet to disk. Since encode outputs an