Re: parallelizing ghc

2012-02-27 Thread Simon Marlow
On 17/02/2012 18:12, Evan Laforge wrote: Sure, except that if the server is to be used by multiple clients, you will get clashes in the PIT when say two clients both try to compile a module with the same name. The PIT is indexed by Module, which is basically the pair (package,modulename), and th

Re: parallelizing ghc

2012-02-17 Thread Evan Laforge
> Sure, except that if the server is to be used by multiple clients, you will > get clashes in the PIT when say two clients both try to compile a module > with the same name. > > The PIT is indexed by Module, which is basically the pair > (package,modulename), and the package for the main program i

Re: parallelizing ghc

2012-02-17 Thread Simon Marlow
On 17/02/2012 01:59, Evan Laforge wrote: However, the GHC API doesn't provide a way to do this directly (I hadn't really thought about this when I suggested the idea before, sorry). The GHC API provides support for compiling multiple modules in the way that GHCi and --make work; each module is a

Re: parallelizing ghc

2012-02-16 Thread Evan Laforge
> However, the GHC API doesn't provide a way to do this directly (I hadn't > really thought about this when I suggested the idea before, sorry).  The GHC > API provides support for compiling multiple modules in the way that GHCi and > --make work; each module is added to the HPT as it is compiled.

Re: parallelizing ghc

2012-02-13 Thread Simon Marlow
On 10/02/2012 08:01, Evan Laforge wrote: I like the idea! And it should be possible to build this without modifying GHC at all, on top of the GHC API. As you say, you'll need a server process, which accepts command lines, executes them, and sends back the results. A local socket should be fine

Re: parallelizing ghc

2012-02-10 Thread Evan Laforge
> I like the idea!  And it should be possible to build this without modifying > GHC at all, on top of the GHC API.  As you say, you'll need a server > process, which accepts command lines, executes them, and sends back the > results.  A local socket should be fine (and will work on both Unix and >

Re: parallelizing ghc

2012-01-29 Thread Neil Mitchell
Hi Simon, I have found that a factor of 2 parallelism is required on Linux to draw with ghc --make. In particular: GHC --make = 7.688 Shake -j1 = 11.828 (of which 11.702 is spent running system commands) Shake full -j4 = 7.414 (of which 12.906 is spent running system commands) This is for a Hask

Re: parallelizing ghc

2012-01-27 Thread Simon Marlow
On 26/01/2012 23:37, Evan Laforge wrote: I'm slightly surprised by this - in my experience parallel builds beat --make as long as the parallelism is a factor of 2 or more. Is your dependency graph very narrow, or do you have lots of very small modules? I get full parallelism, 4 threads at once

Re: parallelizing ghc

2012-01-26 Thread Nathan Howell
On Thu, Jan 26, 2012 at 3:44 PM, Evan Laforge wrote: > I'd think apple would care about linker performance... I'm even a > little surprised Xcode doesn't have something better than a lightly > hacked gnu ld. Someone mentioned that it was on their wish-list at LLVM 2010 conference... it's hinted

Re: parallelizing ghc

2012-01-26 Thread Evan Laforge
> I've been toying with building my own ld replacement.  I don't know > anything about linkers, but I'd say at least even odds that I can do > better than this. I'm guessing linkers are hard, but gold proves that if you keep the scope small and use modern techniques you can get really good improve

Re: parallelizing ghc

2012-01-26 Thread Evan Laforge
> I'm slightly surprised by this - in my experience parallel builds beat > --make as long as the parallelism is a factor of 2 or more.  Is your > dependency graph very narrow, or do you have lots of very small modules? I get full parallelism, 4 threads at once on a 2 core machine * 2 hyperthread/w

Re: parallelizing ghc

2012-01-26 Thread John Lato
> From: Evan Laforge > > On Wed, Jan 25, 2012 at 11:42 AM, Ryan Newton wrote: >>> package list for me. ?The time is going to be dominated by linking, >>> which is single threaded anyway, so either way works. >> >> What is the state of incremental linkers? ?I thought those existed now. > > I think

Re: parallelizing ghc

2012-01-26 Thread Simon Marlow
On 24/01/2012 03:53, Evan Laforge wrote: > I recently switched from ghc --make to a parallelized build system. I > was looking forward to faster builds, and while they are much faster > at figuring out what has to be rebuilt (which is most of the time for > a small rebuild, since ld dominates), c

Re: parallelizing ghc

2012-01-25 Thread Evan Laforge
On Wed, Jan 25, 2012 at 11:42 AM, Ryan Newton wrote: >> package list for me.  The time is going to be dominated by linking, >> which is single threaded anyway, so either way works. > > What is the state of incremental linkers?  I thought those existed now. I think in some specific cases. I've he

Re: parallelizing ghc

2012-01-25 Thread Ryan Newton
> > package list for me. The time is going to be dominated by linking, > which is single threaded anyway, so either way works. > What is the state of incremental linkers? I thought those existed now. ___ Glasgow-haskell-users mailing list Glasgow-haske

Re: parallelizing ghc

2012-01-24 Thread Mikhail Glushenkov
Hi, On Tue, Jan 24, 2012 at 7:04 PM, Evan Laforge wrote: >> I'm also interested in a "build server" mode for ghc. I have written a >> parallel wrapper for 'ghc --make' [1], but the speed gains are not as >> impressive [2] as I hoped because of the duplicated work. > > Was the duplicated work rere

Re: parallelizing ghc

2012-01-24 Thread Evan Laforge
> One immediate problem I see with this is linking - 'ghc --make > Main.hs' is able to figure out what packages a program depends on, > while 'ghc Main.o ... -o Main' requires the user to specify them > manually with -package. So you'll either need to pass this information > back to the parent proc

Re: parallelizing ghc

2012-01-24 Thread Mikhail Glushenkov
Hi, On Tue, Jan 24, 2012 at 4:53 AM, Evan Laforge wrote: > So ghc --make provides two things: a dependency chaser and a way to > keep the compiler resident as it compiles new files.  Since the > dependency chaser will never be as powerful as a real build system, it > occurs to me that the only re

Re: parallelizing ghc

2012-01-24 Thread Mikhail Glushenkov
Hi, On Tue, Jan 24, 2012 at 4:53 AM, Evan Laforge wrote: > [...] > > So ghc --make provides two things: a dependency chaser and a way to > keep the compiler resident as it compiles new files. Since the > dependency chaser will never be as powerful as a real build system, it > occurs to me that t