Re: [Haskell-cafe] concurrency vs. I/O in GHC
On Wed, Oct 27, 2010 at 23:09, Andrew Coppin andrewcop...@btinternet.com wrote: On 27/10/2010 05:00 PM, John Lato wrote: I am somewhat surprised that all capabilities must be ready for GC; I thought with the parallel GC that wouldn't be necessary. But I don't know much about GC implementations so I try not to let their behavior surprise me too much. GHC has a _parallel_ GC implementation, meaning that the GC event runs in parallel on several cores. But it does not (yet) have _concurrent_ GC, meaning that a GC event can happen at the same time as Haskell threads are running. (Basically, which Haskell code running, the references between objects could change while the GC engine is trying to analyse them, which would be Bad.) I understand that the developers are actively working on fixing this, since it can sometimes have a significant effect on the performance of multicore programs... I thought that young generations could be GC'ed while other threads were running, while collecting the old generation required synchronizing all threads. This seems to be what is shown on http://hackage.haskell.org/trac/ghc/blog/new-gc-preview as well. Erik ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On 28/10/2010 09:25 AM, Erik Hesselink wrote: On Wed, Oct 27, 2010 at 23:09, Andrew Coppin andrewcop...@btinternet.com wrote: GHC has a _parallel_ GC implementation, meaning that the GC event runs in parallel on several cores. But it does not (yet) have _concurrent_ GC, meaning that a GC event can happen at the same time as Haskell threads are running. (Basically, which Haskell code running, the references between objects could change while the GC engine is trying to analyse them, which would be Bad.) I understand that the developers are actively working on fixing this, since it can sometimes have a significant effect on the performance of multicore programs... I thought that young generations could be GC'ed while other threads were running, while collecting the old generation required synchronizing all threads. This seems to be what is shown on http://hackage.haskell.org/trac/ghc/blog/new-gc-preview as well. I expect it to land in GHC HEAD in a few months time, and it should be in the autumn 2011 major release of GHC. In other words, this isn't how GHC works now, it's how some future version will work. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On Thu, Oct 28, 2010 at 19:06, Andrew Coppin andrewcop...@btinternet.com wrote: On 28/10/2010 09:25 AM, Erik Hesselink wrote: On Wed, Oct 27, 2010 at 23:09, Andrew Coppin andrewcop...@btinternet.com wrote: GHC has a _parallel_ GC implementation, meaning that the GC event runs in parallel on several cores. But it does not (yet) have _concurrent_ GC, meaning that a GC event can happen at the same time as Haskell threads are running. (Basically, which Haskell code running, the references between objects could change while the GC engine is trying to analyse them, which would be Bad.) I understand that the developers are actively working on fixing this, since it can sometimes have a significant effect on the performance of multicore programs... I thought that young generations could be GC'ed while other threads were running, while collecting the old generation required synchronizing all threads. This seems to be what is shown on http://hackage.haskell.org/trac/ghc/blog/new-gc-preview as well. I expect it to land in GHC HEAD in a few months time, and it should be in the autumn 2011 major release of GHC. In other words, this isn't how GHC works now, it's how some future version will work. Ah, sorry, I missed that. Erik ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On 27/10/2010 05:00 PM, John Lato wrote: I am somewhat surprised that all capabilities must be ready for GC; I thought with the parallel GC that wouldn't be necessary. But I don't know much about GC implementations so I try not to let their behavior surprise me too much. GHC has a _parallel_ GC implementation, meaning that the GC event runs in parallel on several cores. But it does not (yet) have _concurrent_ GC, meaning that a GC event can happen at the same time as Haskell threads are running. (Basically, which Haskell code running, the references between objects could change while the GC engine is trying to analyse them, which would be Bad.) I understand that the developers are actively working on fixing this, since it can sometimes have a significant effect on the performance of multicore programs... ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On 23/10/10 23:17, Donn Cave wrote:
Quoth Claude Heiland-Allenclaudiusmaxi...@goto10.org,
...
The conclusion I drew was that unsafe foreign functions block the
current capability (OS thread) and any threads (Haskell forkIO etc)
currently scheduled on that capability, but other capabilities and
threads continue executing as normal.
... until GC time when all capabilities must be ready. (?)
If a trivial test program would help, here I call the sleep() function,
which I believe on a POSIX platform suspends the thread until receipt
of a SIGALRM.
I wrote a program which shows some interesting behaviour:
8
{-# LANGUAGE ForeignFunctionInterface #-}
module Main (main) where
import GHC.Conc (forkOnIO, numCapabilities)
import Control.Concurrent (threadDelay)
import Foreign.C (CInt)
import System.Environment (getArgs)
foreign import ccall unsafe sleep sleep :: CInt - IO CInt
delayer :: Int - IO ()
delayer n = do
print (delayer, n)
threadDelay 10 -- 10Hz
delayer n
sleeper :: Int - IO ()
sleeper n = do
print (sleeper, n)
_ - sleep 1 -- 1Hz
sleeper n
main :: IO ()
main = do
m - (read . head) `fmap` getArgs
mapM_ (\n - forkOnIO n $ delayer n) [1 .. numCapabilities]
mapM_ (\n - forkOnIO n $ sleeper n) [1 .. numCapabilities - m]
threadDelay 1 -- 100s
8
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 6.12.3
$ uname -a
Linux zebimus 2.6.32-25-generic #44-Ubuntu SMP Fri Sep 17 20:05:27 UTC
2010 x86_64 GNU/Linux
$ ghc -O2 -Wall -threaded --make DelayedSleep.hs
$ ./DelayedSleep +RTS -N4 -S -RTS 3
[snip]
8
By interesting I mean there is lots of output from the delayer threads
on capabilities without sleeper threads (as you would expect), with the
delayer threads on capabilities also having sleeper threads being much
less frequent (as you might also expect). But then there are some long
pauses where there is no output from any thread: my hypothesis is that
the whole runtime is blocked waiting for all threads to be ready for GC
(because +RTS -S shows some GC stats after the end of those pauses).
Claude
--
http://claudiusmaximus.goto10.org
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On 10/23/10 7:54 AM, John Lato wrote: On Fri, Oct 22, 2010 at 6:16 PM, Bulat Ziganshin bulat.zigans...@gmail.com mailto:bulat.zigans...@gmail.com wrote: Hello John, Monday, October 18, 2010, 8:15:42 PM, you wrote: If anyone is listening, I would very much like for there to be a mechanism by which external functions can be called unsafe-ly, but without blocking all other Haskell threads. I have code that does this: +RTS -N2 This doesn't work, which was why the OP asked in the first place. When a thread calls an unsafe foreign function, it blocks everything until that function returns. Is that true? The last time we discussed this in Haskell Cafe the conclusion I drew from the discussion was that unsafe foreign functions block the current thread but not any other thread. Cheers, Greg ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/22/10 19:16 , Bulat Ziganshin wrote: Monday, October 18, 2010, 8:15:42 PM, you wrote: If anyone is listening, I would very much like for there to be a mechanism by which external functions can be called unsafe-ly, but without blocking all other Haskell threads. I have code that does this: +RTS -N2 I think they mean please don't conflate `reentrant' with `blocking' in the FFI. - -- brandon s. allbery [linux,solaris,freebsd,perl] allb...@kf8nh.com system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu electrical and computer engineering, carnegie mellon university KF8NH -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzDKmIACgkQIn7hlCsL25WHMwCgktq4XC3Exdij33maBxN9Vu8p jlcAoJhasAIqVbSYo79z+IrY3zp9GVOR =ZmOh -END PGP SIGNATURE- ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On 23/10/10 17:42, Gregory Crosswhite wrote: On 10/23/10 7:54 AM, John Lato wrote: On Fri, Oct 22, 2010 at 6:16 PM, Bulat Ziganshin bulat.zigans...@gmail.com mailto:bulat.zigans...@gmail.com wrote: Hello John, Monday, October 18, 2010, 8:15:42 PM, you wrote: If anyone is listening, I would very much like for there to be a mechanism by which external functions can be called unsafe-ly, but without blocking all other Haskell threads. I have code that does this: +RTS -N2 This doesn't work, which was why the OP asked in the first place. When a thread calls an unsafe foreign function, it blocks everything until that function returns. Is that true? The last time we discussed this in Haskell Cafe the conclusion I drew from the discussion was that unsafe foreign functions block the current thread but not any other thread. The conclusion I drew was that unsafe foreign functions block the current capability (OS thread) and any threads (Haskell forkIO etc) currently scheduled on that capability, but other capabilities and threads continue executing as normal. Thanks, Claude -- http://claudiusmaximus.goto10.org ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On 10/23/10 12:57 PM, Claude Heiland-Allen wrote: On 23/10/10 17:42, Gregory Crosswhite wrote: On 10/23/10 7:54 AM, John Lato wrote: On Fri, Oct 22, 2010 at 6:16 PM, Bulat Ziganshin This doesn't work, which was why the OP asked in the first place. When a thread calls an unsafe foreign function, it blocks everything until that function returns. Is that true? The last time we discussed this in Haskell Cafe the conclusion I drew from the discussion was that unsafe foreign functions block the current thread but not any other thread. The conclusion I drew was that unsafe foreign functions block the current capability (OS thread) and any threads (Haskell forkIO etc) currently scheduled on that capability, but other capabilities and threads continue executing as normal. Yes, that is what I really meant to say; thank you for using the correct words. :-) Cheers, Greg ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
Quoth Claude Heiland-Allen claudiusmaxi...@goto10.org,
...
The conclusion I drew was that unsafe foreign functions block the
current capability (OS thread) and any threads (Haskell forkIO etc)
currently scheduled on that capability, but other capabilities and
threads continue executing as normal.
If a trivial test program would help, here I call the sleep() function,
which I believe on a POSIX platform suspends the thread until receipt
of a SIGALRM.
If unsafe, during the execution of sleep() in one thread, Haskell
execution will be blocked in the other, so they will alternate.
If safe, the two sleep intervals will overlap. I believe we all
now expect that, but if it does come as a surprise, I hope someone
will test it on a more common platform. +RTS -N2 makes no difference.
Donn Cave, d...@avvanta.com
---
{-# LANGUAGE ForeignFunctionInterface #-}
module Main (main) where
import Control.Concurrent (forkOS)
import Foreign
import Foreign.C
foreign import ccall unsafe sleep sleep :: CInt - IO CInt
rep :: (CInt - IO CInt) - CInt - IO ()
rep f s = do
print s
f s
rep f s
main = do
forkOS $ rep sleep 3
rep sleep 1
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On 10/23/10 2:33 PM, Brandon S Allbery KF8NH wrote: I think they mean please don't conflate `reentrant' with `blocking' in the FFI. Not knowing much about the guts of GHC's implementation of the FFI, I wonder if there would actually be an implementational difference in distinguishing blocking calls vs calls with (potentially) callbacks, or if it's just a terminology problem. Also, we should be distinguishing between C functions which are non-reentrant ---and so we'd want to block all other GHC threads from calling it before the first invocation returns[1]---, vs when the GHC runtime doesn't get itself into a reenterable state before invoking foreign calls. [1] Presumably this should be considered a library problem (e.g., use locks) rather than an FFI problem per se. -- Live well, ~wren ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
On Oct 23, 2010, at 2:57 PM, Claude Heiland-Allen claudiusmaxi...@goto10.org wrote: On 23/10/10 17:42, Gregory Crosswhite wrote: On 10/23/10 7:54 AM, John Lato wrote: On Fri, Oct 22, 2010 at 6:16 PM, Bulat Ziganshin bulat.zigans...@gmail.com mailto:bulat.zigans...@gmail.com wrote: Hello John, Monday, October 18, 2010, 8:15:42 PM, you wrote: If anyone is listening, I would very much like for there to be a mechanism by which external functions can be called unsafe-ly, but without blocking all other Haskell threads. I have code that does this: +RTS -N2 This doesn't work, which was why the OP asked in the first place. When a thread calls an unsafe foreign function, it blocks everything until that function returns. Is that true? The last time we discussed this in Haskell Cafe the conclusion I drew from the discussion was that unsafe foreign functions block the current thread but not any other thread. The conclusion I drew was that unsafe foreign functions block the current capability (OS thread) and any threads (Haskell forkIO etc) currently scheduled on that capability, but other capabilities and threads continue executing as normal. I haven't tested it recently, but I think that is mostly correct except that an unsafe call will also prevent major collections, which will eventually tie up the rest of the capabilities as well. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
From: Donn Cave d...@avvanta.com [I wrote initially, ...] As I have migrated more of my application into Haskell, I find that I/O in one thread effectively blocks other threads. Resolved - the SSL_read external I/O function needs to be safe. If anyone is listening, I would very much like for there to be a mechanism by which external functions can be called unsafe-ly, but without blocking all other Haskell threads. I have code that does this: calcUpdate a b c = do expensiveComputation a b expensiveComputation a c expensiveComputation mutates the second argument (b or c here), but does not call back into Haskell or update a and I would very much like to run these two in parallel, however it's not possible if expensiveComputation is unsafe. If I make expensiveComputation safe, the extra time necessary for safe-ness outweighs the gains from parallelization for even reasonably-sized inputs (the expensiveComputation is called 100k's of times or more). My current plan is to handle the parallelization in C, but I'd prefer to do it in Haskell if possible. John ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
Donn Cave d...@avvanta.com writes: As I have migrated more of my application into Haskell, I find that I/O in one thread effectively blocks other threads. That's rather the opposite of what I need - I don't care so much if Haskell threads manage to compute in parallel, but the application should continue to function, in other threads, while an I/O has one thread blocked. Is that supposed to happen, is there anything I can do about it? Dumb question: are you compiling -threaded? G -- Gregory Collins g...@gregorycollins.net ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] concurrency vs. I/O in GHC
[I wrote initially, ...] As I have migrated more of my application into Haskell, I find that I/O in one thread effectively blocks other threads. Resolved - the SSL_read external I/O function needs to be safe. My apologies for botching the mailing list threading, but I deleted the mail wherein someone helpfully asked whether I'd made my foreign calls safe or unsafe, and I can't see that message on the archives. My initial take was, yes, of course I make external calls safe, because they sure are going to trigger callbacks into the Haskell runtime and the documentation is quite clear about that. However, I'd made SSL_read unsafe, since no callbacks there. Reversed that, and now my threads execute concurrently - SSL_read doesn't block Haskell execution in all the other threads. Either way, I can also report that it doesn't seem to matter whether SSL_read was called from a callback, or not. If unsafe, it blocks all the other threads either way, and if safe, it doesn't block them either way. So, thanks a lot, whoever you were! Henceforth I will play it safe and eschew unsafeness. I vaguely recall a discussion here a ways back, wherein the terms safe and unsafe were lamented and various confusing propositions were bandied about. I should dig that up if I can, and see if this practical example sheds some light on any of it. Donn Cave, d...@avvanta.com ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
