Saurabh Nanda <[email protected]> writes: >> >> Did you ever make any progress on this, Saurabh? >> > > We made progress in some sense, by introducing a separate `stack build -j1` > step in our CI pipeline for compiling packages that are known to use a lot > of memory. > > >> >> * -j just tells GHC to parallelise compilation across modules. This can >> increase the maximum heap size needed by the compiler. >> > > > From the docs, it wasn't very clear to me how -j interacts with -M when > both the options are passed to the GHC process. Is it the max heap size > across all build, or per build? > The short answer is that they don't interact: -j is a GHC flag whereas -M is an RTS flag. -M controls the amount of heap that the RTS allows the mutator (GHC in this case) to allocate. This includes all threads. GHC when run with -j is just like any other threaded Haskell program.
>> * -M is a bit tricky to define. For one, it defines the maximum heap >> size beyond which we will terminate. However, we also use it in >> garbage collector to make various decisions about GC scheduling. I'll >> admit that I'm not terribly familiar with the details here. >> >> Note that -M does not guarantee that GHC will find a way to keep your >> program under the limit that you provide. It merely ensures that the >> program doesn't exceed the given size, aborting if necessary. >> > > > Quoting from > https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M > : > > *> The maximum heap size also affects other garbage collection parameters: > when the amount of live data in the heap exceeds a certain fraction of the > maximum heap size, compacting collection will be automatically enabled for > the oldest generation, and the -F parameter will be reduced in order to > avoid exceeding the maximum heap size.* > > It just makes it sound that the RTS is going to tweak the GC algo, and the > number of time GC is run, to avoid crossing the heap limit. However, I've > found the GHC process easily consuming more memory than what is specified > in the -M flag (as reported by top). > Yes, as I mentioned we do tweak some things in the GC; however, these tweaks are really a best-effort attempt to avoid going over the limit. It's entirely possible that your mutator will be terminated if it wants to use significantly more than the limit set with -M. There is relatively little else the RTS can do in this case that wouldn't require explicit cooperation from the mutator to keep working sets down. Cheers, - Ben
signature.asc
Description: PGP signature
_______________________________________________ ghc-devs mailing list [email protected] http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
