Re: [Haskell-cafe] Alternative to Data.Binary
Don Stewart ha scritto: grzegorz.chrupala: Hi all, Is there a serialization library other than the Data.Binary from hackage? I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow. [...] Have you tried the latest release, which modified the Map and [a] instances? Tried right now. My [UArr (Word32 :*: Word8)], where the list length is 17770, now requires 660 MB of memory, when decoding, against 840 MB with the previous version of the binary package. This is fantastic, thanks! Regards Manlio ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Alternative to Data.Binary
Don Stewart-2 wrote: > > Have you tried the latest release, which modified the Map and [a] > instances? > No, I'm working with 0.5. I'll give the new version a try. Thanks! -- Grzegorz -- View this message in context: http://www.nabble.com/Alternative-to-Data.Binary-tp22512229p22514771.html Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Alternative to Data.Binary
grzegorz.chrupala: > > Hi all, > Is there a serialization library other than the Data.Binary from hackage? > > I am using Data.Binary in a couple of projects, but I have found its stack > and memory usage very hard to control. Its very common that decoding a map > or list of non-trivial size uses up all available RAM, or causes a stack > overflow. > > I don't care that much about it being extremely fast, I just want to stop > worrying that if I try to read a file a few percent larger than the last > time, my program will suddenly stop working. Have you tried the latest release, which modified the Map and [a] instances? -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Alternative to Data.Binary
Svein Ove Aas ha scritto: On Sat, Mar 14, 2009 at 1:37 PM, Grzegorz Chrupala wrote: Hi all, Is there a serialization library other than the Data.Binary from hackage? I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow. That little problem appears to be an artifact of the particular Binary instance for lists (the map instance starts by converting to/from a list), and should be fixable in principle, for example by writing them in chunks of 256 elements. I can confirm that reading serialized UArr from uvector package does not cause memory problems. Manlio ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Alternative to Data.Binary
On Sat, Mar 14, 2009 at 1:37 PM, Grzegorz Chrupala wrote: > > Hi all, > Is there a serialization library other than the Data.Binary from hackage? > > I am using Data.Binary in a couple of projects, but I have found its stack > and memory usage very hard to control. Its very common that decoding a map > or list of non-trivial size uses up all available RAM, or causes a stack > overflow. > That little problem appears to be an artifact of the particular Binary instance for lists (the map instance starts by converting to/from a list), and should be fixable in principle, for example by writing them in chunks of 256 elements. That said, the current instance really should only have problems when *writing*, in that it'll force the entire list to figure out its length before it starts writing any of it. That this is not, in fact, the case suggests that the reader should be fixable without changing its on-disk format. -- Svein Ove Aas ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Alternative to Data.Binary
Hi all, Is there a serialization library other than the Data.Binary from hackage? I am using Data.Binary in a couple of projects, but I have found its stack and memory usage very hard to control. Its very common that decoding a map or list of non-trivial size uses up all available RAM, or causes a stack overflow. I don't care that much about it being extremely fast, I just want to stop worrying that if I try to read a file a few percent larger than the last time, my program will suddenly stop working. Best, -- Grzegorz -- View this message in context: http://www.nabble.com/Alternative-to-Data.Binary-tp22512229p22512229.html Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
