Re: parallelizing ghc

2012-02-27 Thread Simon Marlow
On 17/02/2012 18:12, Evan Laforge wrote: Sure, except that if the server is to be used by multiple clients, you will get clashes in the PIT when say two clients both try to compile a module with the same name. The PIT is indexed by Module, which is basically the pair (package,modulename), and

Re: parallelizing ghc

2012-02-17 Thread Simon Marlow
On 17/02/2012 01:59, Evan Laforge wrote: However, the GHC API doesn't provide a way to do this directly (I hadn't really thought about this when I suggested the idea before, sorry). The GHC API provides support for compiling multiple modules in the way that GHCi and --make work; each module is

Re: parallelizing ghc

2012-02-17 Thread Evan Laforge
Sure, except that if the server is to be used by multiple clients, you will get clashes in the PIT when say two clients both try to compile a module with the same name. The PIT is indexed by Module, which is basically the pair (package,modulename), and the package for the main program is

Re: parallelizing ghc

2012-02-16 Thread Evan Laforge
However, the GHC API doesn't provide a way to do this directly (I hadn't really thought about this when I suggested the idea before, sorry).  The GHC API provides support for compiling multiple modules in the way that GHCi and --make work; each module is added to the HPT as it is compiled.  

Re: parallelizing ghc

2012-02-13 Thread Simon Marlow
On 10/02/2012 08:01, Evan Laforge wrote: I like the idea! And it should be possible to build this without modifying GHC at all, on top of the GHC API. As you say, you'll need a server process, which accepts command lines, executes them, and sends back the results. A local socket should be

Re: parallelizing ghc

2012-01-29 Thread Neil Mitchell
Hi Simon, I have found that a factor of 2 parallelism is required on Linux to draw with ghc --make. In particular: GHC --make = 7.688 Shake -j1 = 11.828 (of which 11.702 is spent running system commands) Shake full -j4 = 7.414 (of which 12.906 is spent running system commands) This is for a

Re: parallelizing ghc

2012-01-27 Thread Simon Marlow
On 26/01/2012 23:37, Evan Laforge wrote: I'm slightly surprised by this - in my experience parallel builds beat --make as long as the parallelism is a factor of 2 or more. Is your dependency graph very narrow, or do you have lots of very small modules? I get full parallelism, 4 threads at

Re: parallelizing ghc

2012-01-26 Thread Simon Marlow
On 24/01/2012 03:53, Evan Laforge wrote: I recently switched from ghc --make to a parallelized build system. I was looking forward to faster builds, and while they are much faster at figuring out what has to be rebuilt (which is most of the time for a small rebuild, since ld dominates),

Re: parallelizing ghc

2012-01-26 Thread John Lato
From: Evan Laforge qdun...@gmail.com On Wed, Jan 25, 2012 at 11:42 AM, Ryan Newton rrnew...@gmail.com wrote: package list for me. ?The time is going to be dominated by linking, which is single threaded anyway, so either way works. What is the state of incremental linkers? ?I thought those

Re: parallelizing ghc

2012-01-26 Thread Evan Laforge
I'm slightly surprised by this - in my experience parallel builds beat --make as long as the parallelism is a factor of 2 or more.  Is your dependency graph very narrow, or do you have lots of very small modules? I get full parallelism, 4 threads at once on a 2 core machine * 2

Re: parallelizing ghc

2012-01-26 Thread Nathan Howell
On Thu, Jan 26, 2012 at 3:44 PM, Evan Laforge qdun...@gmail.com wrote: I'd think apple would care about linker performance... I'm even a little surprised Xcode doesn't have something better than a lightly hacked gnu ld. Someone mentioned that it was on their wish-list at LLVM 2010

Re: parallelizing ghc

2012-01-25 Thread Evan Laforge
On Wed, Jan 25, 2012 at 11:42 AM, Ryan Newton rrnew...@gmail.com wrote: package list for me.  The time is going to be dominated by linking, which is single threaded anyway, so either way works. What is the state of incremental linkers?  I thought those existed now. I think in some specific

Re: parallelizing ghc

2012-01-24 Thread Mikhail Glushenkov
Hi, On Tue, Jan 24, 2012 at 4:53 AM, Evan Laforge qdun...@gmail.com wrote: [...] So ghc --make provides two things: a dependency chaser and a way to keep the compiler resident as it compiles new files. Since the dependency chaser will never be as powerful as a real build system, it occurs

Re: parallelizing ghc

2012-01-24 Thread Mikhail Glushenkov
Hi, On Tue, Jan 24, 2012 at 4:53 AM, Evan Laforge qdun...@gmail.com wrote: So ghc --make provides two things: a dependency chaser and a way to keep the compiler resident as it compiles new files.  Since the dependency chaser will never be as powerful as a real build system, it occurs to me

Re: parallelizing ghc

2012-01-24 Thread Evan Laforge
One immediate problem I see with this is linking - 'ghc --make Main.hs' is able to figure out what packages a program depends on, while 'ghc Main.o ... -o Main' requires the user to specify them manually with -package. So you'll either need to pass this information back to the parent process,

Re: parallelizing ghc

2012-01-24 Thread Mikhail Glushenkov
Hi, On Tue, Jan 24, 2012 at 7:04 PM, Evan Laforge qdun...@gmail.com wrote: I'm also interested in a build server mode for ghc. I have written a parallel wrapper for 'ghc --make' [1], but the speed gains are not as impressive [2] as I hoped because of the duplicated work. Was the duplicated

parallelizing ghc

2012-01-23 Thread Evan Laforge
I recently switched from ghc --make to a parallelized build system. I was looking forward to faster builds, and while they are much faster at figuring out what has to be rebuilt (which is most of the time for a small rebuild, since ld dominates), compilation of the whole system is either the same