Re: [Haskell-cafe] Long running Haskell program
Also, a *service* should have a persistence periodical action which should evaluate (most part of) the state thunks in their values to be serialized. This work for the structure similarity of NFData and Show/Binary classes. When there is a not directly serializable part of the state , things can get complicate, but resolving the persistence issue for those parts should resolve also the space leak, I think. paolino 2009/11/11 Paolino paolo.verone...@gmail.com Hello leimy, the only simple solution I have found to avoid a leaking state of a server is doing a periodical rnf of it, this implying the NFData constraint on its datatype. The reader should leak only if you nest forever the local function. paolino 2009/11/11 David Leimbach leim...@gmail.com As some of you may know, I've been writing commercial Haskell code for a little bit here (about a year and a half) and I've just recently had to write some code that was going to run have to run for a really long time before being restarted, possibly months or years if all other parts of the system cooperate as it's part of a server infrastructure management system. I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) Anyway, rather than try to paste it all here with images and such I thought I'd stick it up on my blog so others could maybe benefit from the anecdote. It's difficult to disclose enough useful information as it is commercial code not under an open source license, but there's neat diagrams and stuff there so hopefully the colors are at least amusing :-) http://leimy9.blogspot.com/2009/11/long-running-haskell-applications.html Dave ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Long running Haskell program
As some of you may know, I've been writing commercial Haskell code for a little bit here (about a year and a half) and I've just recently had to write some code that was going to run have to run for a really long time before being restarted, possibly months or years if all other parts of the system cooperate as it's part of a server infrastructure management system. I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) Anyway, rather than try to paste it all here with images and such I thought I'd stick it up on my blog so others could maybe benefit from the anecdote. It's difficult to disclose enough useful information as it is commercial code not under an open source license, but there's neat diagrams and stuff there so hopefully the colors are at least amusing :-) http://leimy9.blogspot.com/2009/11/long-running-haskell-applications.html Dave ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
On Wed, Nov 11, 2009 at 7:43 AM, David Leimbach leim...@gmail.com wrote: I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) It sounds to me like you were storing a Map in a StateT. Since the usual State and StateT monads don't force the evaluation of their payload, I'm not terribly surprised that such a leak should arise. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
Is there a state monad that is strict on the state but lazy on the computation? Of course, strictness in the state will force a portion of the computation to be run, but there may be significant portions of it which are not run. Would there be a way to write a state monad such that it is entirely lazy, but then to wrap either the computation or the state in an 'eager' strategy datatype which takes care of this in a more flexible manner? Thanks, Matthew 2009/11/11 Bryan O'Sullivan b...@serpentine.com On Wed, Nov 11, 2009 at 7:43 AM, David Leimbach leim...@gmail.com wrote: I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) It sounds to me like you were storing a Map in a StateT. Since the usual State and StateT monads don't force the evaluation of their payload, I'm not terribly surprised that such a leak should arise. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
On Wed, Nov 11, 2009 at 1:09 PM, Matthew Pocock
matthew.poc...@ncl.ac.uk wrote:
Is there a state monad that is strict on the state but lazy on the
computation? Of course, strictness in the state will force a portion of the
computation to be run, but there may be significant portions of it which are
not run. Would there be a way to write a state monad such that it is
entirely lazy, but then to wrap either the computation or the state in an
'eager' strategy datatype which takes care of this in a more flexible
manner?
I think replacing put s with put $! s should guarantee that the
state is evaluated.
If you're using get and put in many place in the code, you could try
something along these lines:
newtype SStateT s m a = S { unS :: StateT s m a } deriving (Monad, etc.)
instance (Monad m) = MonadState s (SStateT s m) where
get = S get
put s = S (put $! s)
--
Dave Menendez d...@zednenem.com
http://www.eyrie.org/~zednenem/
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
Hello leimy, the only simple solution I have found to avoid a leaking state of a server is doing a periodical rnf of it, this implying the NFData constraint on its datatype. The reader should leak only if you nest forever the local function. paolino 2009/11/11 David Leimbach leim...@gmail.com As some of you may know, I've been writing commercial Haskell code for a little bit here (about a year and a half) and I've just recently had to write some code that was going to run have to run for a really long time before being restarted, possibly months or years if all other parts of the system cooperate as it's part of a server infrastructure management system. I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) Anyway, rather than try to paste it all here with images and such I thought I'd stick it up on my blog so others could maybe benefit from the anecdote. It's difficult to disclose enough useful information as it is commercial code not under an open source license, but there's neat diagrams and stuff there so hopefully the colors are at least amusing :-) http://leimy9.blogspot.com/2009/11/long-running-haskell-applications.html Dave ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
On Wed, Nov 11, 2009 at 9:51 AM, Bryan O'Sullivan b...@serpentine.comwrote: On Wed, Nov 11, 2009 at 7:43 AM, David Leimbach leim...@gmail.com wrote: I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) It sounds to me like you were storing a Map in a StateT. Since the usual State and StateT monads don't force the evaluation of their payload, I'm not terribly surprised that such a leak should arise. That's exactly what was happening. The system was being far too lazy (by choices I made but didn't fully understand). By pulling the Map out of the state, and pushing it back into the state, as the definition of my looping, things got a lot better. I didn't see another *easy* way to force the state to be evaluated, except by doing IO on intermediate values. seq will only evaluate strictly if it's just underneath something else that's already been evaluated :-). The runtime doesn't look for seqs to force evaluation on. I figured I was better off just creating a dependency in the evaluation, near the outermost portion of the program (the loop) that would cause a strict evaluation, and so far I was right :-) Program behaves very well now, and responds much better too. Dave ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
On Wed, Nov 11, 2009 at 10:29 AM, David Menendez d...@zednenem.com wrote:
On Wed, Nov 11, 2009 at 1:09 PM, Matthew Pocock
matthew.poc...@ncl.ac.uk wrote:
Is there a state monad that is strict on the state but lazy on the
computation? Of course, strictness in the state will force a portion of
the
computation to be run, but there may be significant portions of it which
are
not run. Would there be a way to write a state monad such that it is
entirely lazy, but then to wrap either the computation or the state in an
'eager' strategy datatype which takes care of this in a more flexible
manner?
I think replacing put s with put $! s should guarantee that the
state is evaluated.
If you're using get and put in many place in the code, you could try
something along these lines:
newtype SStateT s m a = S { unS :: StateT s m a } deriving (Monad, etc.)
instance (Monad m) = MonadState s (SStateT s m) where
get = S get
put s = S (put $! s)
That's interesting, and once I have time to come back to this part of the
project (I was behind schedule at this point!) I'll try something like that.
--
Dave Menendez d...@zednenem.com
http://www.eyrie.org/~zednenem/
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
leimy2k: I figured I was better off just creating a dependency in the evaluation, near the outermost portion of the program (the loop) that would cause a strict evaluation, and so far I was right :-) Program behaves very well now, and responds much better too. Do you know if Control.Monad.State.Strict is enough to get the behaviour you need? -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Long running Haskell program
On Wed, Nov 11, 2009 at 11:19 AM, Don Stewart d...@galois.com wrote: leimy2k: I figured I was better off just creating a dependency in the evaluation, near the outermost portion of the program (the loop) that would cause a strict evaluation, and so far I was right :-) Program behaves very well now, and responds much better too. Do you know if Control.Monad.State.Strict is enough to get the behaviour you need? I'll give that a go. Most of my trouble figuring out the space leak has been around identifying what was really responsible for the problem. The functions that were listed as eating the space in -hc runs were not ultimately the ones causing the lack of strictness, in that they would have had to have been evaluated at a higher layer in a non-lazy way. So my take away from all of this is, when you have a space leak in haskell, start from the outer most evaluations inward, not the other way around!!! I think that would have saved me a ton of time. Dave -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
