[Haskell-cafe] Re: Long running Haskell program
David Menendez wrote:
I think replacing put s with put $! s should guarantee that the
state is evaluated.
If you're using get and put in many place in the code, you could try
something along these lines:
newtype SStateT s m a = S { unS :: StateT s m a } deriving (Monad, etc.)
instance (Monad m) = MonadState s (SStateT s m) where
get = S get
put s = S (put $! s)
Interestingly, this is different from Control.Monad.State.Strict . The
latter never forces the state itself, just the pair constructor of the
(result,state) pair.
Here the different cases:
evalLazy m = Control.Monad.State.Lazy.evalState m 0
evalStrict m = Control.Monad.State.Strict.evalState m 0
-- Pair constructor non-bottom
GHCi evalLazy $ put undefined
()
GHCi evalStrict $ put undefined
()
-- Pair constructor bottom
GHCi evalLazy $ put $! undefined
*** Exception: Prelude.undefined
GHCi evalStrict $ put $! undefined
*** Exception: Prelude.undefined
-- Last pair constructor non-bottom
GHCi evalLazy $ (put $! undefined) put 1
()
-- Everything bottom
GHCi evalStrict $ (put $! undefined) put 1
*** Exception: Prelude.undefined
Regards,
apfelmus
--
http://apfelmus.nfshost.com
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Long running Haskell program
2009/11/12 Heinrich Apfelmus apfel...@quantentunnel.de Interestingly, this is different from Control.Monad.State.Strict . The latter never forces the state itself, just the pair constructor of the (result,state) pair. Yes. This bit me the first time I came across it. I think we need a Control. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Long running Haskell program
2009/11/12 Heinrich Apfelmus apfel...@quantentunnel.de Interestingly, this is different from Control.Monad.State.Strict . The latter never forces the state itself, just the pair constructor of the (result,state) pair. Yes. This bit me the first time I came across it. I think we need a Control.Monad.State.StrictOnState with strict behaviour on the state value. I notice this same underlying issue is coming up in more than one thread on these lists. Matthew ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Long running Haskell program
On Thu, Nov 12, 2009 at 8:01 AM, Matthew Pocock matthew.poc...@ncl.ac.uk wrote: Yes. This bit me the first time I came across it. I think we need a Control.Monad.State.StrictOnState with strict behaviour on the state value. I notice this same underlying issue is coming up in more than one thread on these lists. This monad would indeed be useful, however it may be confusing as well. For example, if you are inserting elements in a Map using a lazy insert, seq'ing the map will only reduce it to WHNF (of course) and move the insertion down one level in the tree. IOW, I'd guess that OP's problem can't be solved just by seq'ing the map, he would have to use strict operations. There's a strict insertWith in Data.Map, but IIRC it's strict only on its combining operation, not on the insertion itself. Hmmm... maybe Control.Monad.State.RnfOnState? =P Cheers, -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Long running Haskell program
How about: instance (Monad m) = MonadState s (SStateT s m) where get = S get put s = S (put $ using s $ strategy m) where our state monad has a strategy field? Matthew ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Long running Haskell program
David Leimbach leim...@gmail.com writes: As some of you may know, I've been writing commercial Haskell code for a little bit here (about a year and a half) and I've just recently had to write some code that was going to run have to run for a really long time before being restarted, possibly months or years if all other parts of the system cooperate as it's part of a server infrastructure management system. I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) Anyway, rather than try to paste it all here with images and such I thought I'd stick it up on my blog so others could maybe benefit from the anecdote. It's difficult to disclose enough useful information as it is commercial code not under an open source license, but there's neat diagrams and stuff there so hopefully the colors are at least amusing :-) http://leimy9.blogspot.com/2009/11/long-running-haskell-applications.html Can you copy you blog at here? http://leimy9.blogspot.com/2009/11/long-running-haskell-applications.html is filter by GFW (http://en.wikipedia.org/wiki/Golden_Shield_Project) i can't see it. About crash free program, you can consider multi-process design, keep Simple and Stable core running in RootProcess, and Core module won't crash, and make unstable module running in ChildProcess, if you occur some un-catch exception from ChildProcess, just reboot those sub-module. I'm research some Haskell/Gtk+ program, sometimes un-catch exception is unavoidable, multi-process is good chose to avoid some exception crash all program. Cheers, -- Andy ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Long running Haskell program
On Wed, Nov 11, 2009 at 8:20 AM, Andy Stewart lazycat.mana...@gmail.comwrote: David Leimbach leim...@gmail.com writes: As some of you may know, I've been writing commercial Haskell code for a little bit here (about a year and a half) and I've just recently had to write some code that was going to run have to run for a really long time before being restarted, possibly months or years if all other parts of the system cooperate as it's part of a server infrastructure management system. I recently ran into some serious space leak difficulties that would ultimately cause this program to crash some time after startup (my simulator is also written in Haskell, and runs a LOT faster than the real application ever could, this has enabled me to fast forward a bit the data growth issues and crash in minutes instead of days!) Anyway, rather than try to paste it all here with images and such I thought I'd stick it up on my blog so others could maybe benefit from the anecdote. It's difficult to disclose enough useful information as it is commercial code not under an open source license, but there's neat diagrams and stuff there so hopefully the colors are at least amusing :-) http://leimy9.blogspot.com/2009/11/long-running-haskell-applications.html Can you copy you blog at here? http://leimy9.blogspot.com/2009/11/long-running-haskell-applications.html is filter by GFW (http://en.wikipedia.org/wiki/Golden_Shield_Project) i can't see it. About crash free program, you can consider multi-process design, keep Simple and Stable core running in RootProcess, and Core module won't crash, and make unstable module running in ChildProcess, if you occur some un-catch exception from ChildProcess, just reboot those sub-module. Believe it or not, this is a stack of Erlang - Haskell - C. It works via pipes and is a concurrent system of management. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Long running Haskell program
I used to be a victim of GFW, so I can feel your pain. You may try to subscribe to http://leimy9.blogspot.com/feeds/posts/default in your Google Reader. In case that fails too, I've pasted the blog post below, with no images: I've been using Haskell in a serious way for about 2 years. Been using it in a professional sense about 1.5 years now in that, yes, I am one of the lucky ones that gets to use Haskell at work. The ride has been pretty smooth most of the time, as I've found that the type system especially helps me to rule out certain classes of bugs, easily test rather large chunks of programs as they're pure. The interpreter allows experimentation and iteration of ideas that can then be composed into the final compiled programs. All of this gives me a good deal of confidence that the code I'm writing is correct to some degree up front, something I've come to expect from functional programming languages over the years and greatly appreciate. However I've felt compelled to comment that things aren't always so smooth either. I spent the better part of a weekend and a Monday tracking down a space leak in a program that just was not allowed to leak space. I have a stack of a ReaderT StateT IO that I use to communicate with a device through a passthrough program that speaks CAN to a device for the purposes of creating a serial console connection where there is only a CAN bus. The Haskell program is responsible for the management of the data found at the other end o the serial connection and supports operations to the device through the serial channel via a simple text protocol while forever polling data on the serial endpoint. What I had done was the equivalent of pollerLoop :: Poller () pollerLoop = forever pollOnce Where Poller is my monad stack. pollOnce is defined as, pollOnce :: Poller () pollOnce = do checkCommandChannel -- see if there's a pending command to run executePoll Yes, this application is multi-threaded. I have a logger thread, a thread watching standard input of the program for queries and issuing commands to the endpoint. I have a thread per Poller, and the ability to poll devices simultaneously. The poller includes an implementation of my little expect syntax which was based on a naive implementation of hGetChar and checking for a desired result or timing out eventually. The really data inefficient version is a real screamer, beating the heck out of my Parsec or even ReadP version, but because of the way I wrote it, with a lot of reversing and prefix checking and substring slicing into temporary areas, it's not useful for large input blocks over a long time frame. Still it's so fast that it's appropriate for certain sections of code. I measured and verified this via space profiling to see the real runtime heap utilization (nice feature btw, I'd be dead without it right now I think). So what's the problem you're probably thinking? Well it turns out that since part of my state in StateT is a Data.Map, and that all my polling and parsing of expect-passing blocks caused updates to a Map, coupled with the language's default laziness caused a bit of a bomb of PAP (Partial APplications of functions). I had an ill-timed, project wise, discovery of a space leak. I tried sprinkling $! and seq all over the place, rewriting big chunks of code that used Text.Regex.Posix, to use Parsec and only got incremental improvements. The growth problem still existed, and would eventually exhaust memory. This was a real problem as this was an application that was not supposed to stop when the other conditions of the management system were ok. It could run for months or even years! I went through a whirlwind of emotions, and considered that perhaps I should be working a different job. Perhaps I could be a lion tamer? It turned out what I thought was a lion was really an anteater though... but that's literally a different story. It turns out that by looping not inside the monad, but over the execStateT/runReaderT expression I could pull all the state out, and then re-inject it into another execStateT/runReaderT each poll, which forced the strictness I needed on all state data, made the system respond faster, and best of all, not crash Diagrams below: This first one is the before picture. It shows the data growth by cost center in my code. As you can see things are getting worse almost linearly as I poll. This picture illustrates the result of pulling the state out of the Monad, and re-injecting it, forcing it to be evaluated. As you can see, I've got much more manageable memory utilization. This final one shows the new algorithm running with 2 threads, one spawned a few seconds into the run. You can see the initial burst of the fast but inefficient expect algorithm, followed by a much more regular memory utilization pattern. I'd like to thank all the folks on #haskell on FreeNode who gave me suggestions and hints to my vague problems regarding data growth and
