Re: Building on limited memory environments (or does -M really work?)

[Niklas Hambüchen]

Hey Saurabh,

from my experience with CircleCI it builds on machines with e.g. 32
cores showing up in htop (but allows you to use way less that that).

But ghc sees 32 cores so -j will compile up to 32 modules at the same
time (thus using tons of RAM).

I solved that by setting -jN to the actual number of cores I bought from
CircleCI.

Greetings!
Niklas

On 21/10/17 14:36, Saurabh Nanda wrote:
> We're struggling to get our build pipeline working on CircleCI, which
> has a 4GB RAM limit.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Building on limited memory environments (or does -M really work?)

[Oleg Grenrus]

Note that there are two -j flags, one to `stack` and one to `ghc`.

I'm not completely sure what `stack`s -j does at the moment, but IIRC it
specifies how many *packages* are built in parallel (maybe running tests
is counted towards that job limit too). This means that `stack build
-j3` may (and most likely will) spawn 3 ghc building different packages,
if dependency graph permits. This means that `stack -j3
--ghc-options="+RTS -M1G"` may end up using around 3G of memory.

And as Ben already mentioned, GHC's -j affects "module
parallellisation", i.e. how many modules to build simultaneously.

So there are two `-j`s: stack's affects how many GHCs there are, and
GHC's only how many threads there are. First multiplies -M memory usage,
later doens't.

Another thing to consider, is how both affect performance. And that's
not an easy question to answer. To make situation even more interesting,
latest cabal's support per-component builds, so instead of "how many
parallel packages we build", it's "how many parallel components we
build". As component graph is more granular (e.g. tests and executables
are mostly independent leafs), there is more parallelization
opportunities. I remember reading a discussion mentioning to have `-j
N:M` for "N ghc's building M modules", so you can fine-tune that for
your multi package project. Also there is
https://github.com/haskell/cabal/issues/976 discusses how to make
multiple GHC to co-operate with shared module limit.

Also by digging a bit of Cabal's issue tracker, there are issues like
https://github.com/haskell/cabal/issues/1529. For example linker's
memory usage, and how -j affects that. It might affect you,
where there is memory hungry package building (GHC eats memory), and
also `stack` is linking some executable from other package (also memory
hungry).

As a final note: you can add per package ghc-options in `stack.yaml` [1]
(and cabal.project). At work, for example I have (in `cabal.project`)
few packages with

    package memory-hungry-package
    ghc-options: +RTS -M2G -RTS

where other packages are unrestricted. This wins us a bit in time, as
for most packages GHC's RTS doesn't need to worry about memory usage.

Cheers, Oleg

- [1]
https://docs.haskellstack.org/en/stable/yaml_configuration/#ghc-options

On 08.11.2017 17:27, Ben Gamari wrote:
> Saurabh Nanda  writes:
>
>>> Did you ever make any progress on this, Saurabh?
>>>
>> We made progress in some sense, by introducing a separate `stack build -j1`
>> step in our CI pipeline for compiling packages that are known to use a lot
>> of memory.
>>
>>
>>>  * -j just tells GHC to parallelise compilation across modules. This can
>>> increase the maximum heap size needed by the compiler.
>>>
>>
>> From the docs, it wasn't very clear to me how -j interacts with -M when
>> both the options are passed to the GHC process. Is it the max heap size
>> across all build, or per build?
>>
> The short answer is that they don't interact: -j is a GHC flag whereas
> -M is an RTS flag. -M controls the amount of heap that the RTS allows
> the mutator (GHC in this case) to allocate. This includes all
> threads. GHC when run with -j is just like any other threaded Haskell
> program.
>
>>>  * -M is a bit tricky to define. For one, it defines the maximum heap
>>>size beyond which we will terminate. However, we also use it in
>>>garbage collector to make various decisions about GC scheduling. I'll
>>>admit that I'm not terribly familiar with the details here.
>>>
>>> Note that -M does not guarantee that GHC will find a way to keep your
>>> program under the limit that you provide. It merely ensures that the
>>> program doesn't exceed the given size, aborting if necessary.
>>>
>>
>> Quoting from
>> https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M
>> :
>>
>> *> The maximum heap size also affects other garbage collection parameters:
>> when the amount of live data in the heap exceeds a certain fraction of the
>> maximum heap size, compacting collection will be automatically enabled for
>> the oldest generation, and the -F parameter will be reduced in order to
>> avoid exceeding the maximum heap size.*
>>
>> It just makes it sound that the RTS is going to tweak the GC algo, and the
>> number of time GC is run, to avoid crossing the heap limit. However, I've
>> found the GHC process easily consuming more memory than what is specified
>> in the -M flag (as reported by top).
>>
> Yes, as I mentioned we do tweak some things in the GC; however, these
> tweaks are really a best-effort attempt to avoid going over the limit.
> It's entirely possible that your mutator will be terminated if it wants
> to use significantly more than the limit set with -M. There is
> relatively little else the RTS can do in this case that wouldn't require
> explicit cooperation from the mutator to keep working sets down.
>
> Cheers,
>
> - Ben
>
>
>
> 

Re: Building on limited memory environments (or does -M really work?)

[Ben Gamari]

Saurabh Nanda  writes:

>>
>> Did you ever make any progress on this, Saurabh?
>>
>
> We made progress in some sense, by introducing a separate `stack build -j1`
> step in our CI pipeline for compiling packages that are known to use a lot
> of memory.
>
>
>>
>>  * -j just tells GHC to parallelise compilation across modules. This can
>> increase the maximum heap size needed by the compiler.
>>
>
>
> From the docs, it wasn't very clear to me how -j interacts with -M when
> both the options are passed to the GHC process. Is it the max heap size
> across all build, or per build?
>
The short answer is that they don't interact: -j is a GHC flag whereas
-M is an RTS flag. -M controls the amount of heap that the RTS allows
the mutator (GHC in this case) to allocate. This includes all
threads. GHC when run with -j is just like any other threaded Haskell
program.

>>  * -M is a bit tricky to define. For one, it defines the maximum heap
>>size beyond which we will terminate. However, we also use it in
>>garbage collector to make various decisions about GC scheduling. I'll
>>admit that I'm not terribly familiar with the details here.
>>
>> Note that -M does not guarantee that GHC will find a way to keep your
>> program under the limit that you provide. It merely ensures that the
>> program doesn't exceed the given size, aborting if necessary.
>>
>
>
> Quoting from
> https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M
> :
>
> *> The maximum heap size also affects other garbage collection parameters:
> when the amount of live data in the heap exceeds a certain fraction of the
> maximum heap size, compacting collection will be automatically enabled for
> the oldest generation, and the -F parameter will be reduced in order to
> avoid exceeding the maximum heap size.*
>
> It just makes it sound that the RTS is going to tweak the GC algo, and the
> number of time GC is run, to avoid crossing the heap limit. However, I've
> found the GHC process easily consuming more memory than what is specified
> in the -M flag (as reported by top).
>
Yes, as I mentioned we do tweak some things in the GC; however, these
tweaks are really a best-effort attempt to avoid going over the limit.
It's entirely possible that your mutator will be terminated if it wants
to use significantly more than the limit set with -M. There is
relatively little else the RTS can do in this case that wouldn't require
explicit cooperation from the mutator to keep working sets down.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Building on limited memory environments (or does -M really work?)

[Saurabh Nanda]

>
> Did you ever make any progress on this, Saurabh?
>

We made progress in some sense, by introducing a separate `stack build -j1`
step in our CI pipeline for compiling packages that are known to use a lot
of memory.


>
>  * -j just tells GHC to parallelise compilation across modules. This can
> increase the maximum heap size needed by the compiler.
>


>From the docs, it wasn't very clear to me how -j interacts with -M when
both the options are passed to the GHC process. Is it the max heap size
across all build, or per build?



>  * -M is a bit tricky to define. For one, it defines the maximum heap
>size beyond which we will terminate. However, we also use it in
>garbage collector to make various decisions about GC scheduling. I'll
>admit that I'm not terribly familiar with the details here.
>
> Note that -M does not guarantee that GHC will find a way to keep your
> program under the limit that you provide. It merely ensures that the
> program doesn't exceed the given size, aborting if necessary.
>


Quoting from
https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--M
:

*> The maximum heap size also affects other garbage collection parameters:
when the amount of live data in the heap exceeds a certain fraction of the
maximum heap size, compacting collection will be automatically enabled for
the oldest generation, and the -F parameter will be reduced in order to
avoid exceeding the maximum heap size.*

It just makes it sound that the RTS is going to tweak the GC algo, and the
number of time GC is run, to avoid crossing the heap limit. However, I've
found the GHC process easily consuming more memory than what is specified
in the -M flag (as reported by top).

-- Saurabh.
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

Re: Building on limited memory environments (or does -M really work?)

[Ben Gamari]

Saurabh Nanda  writes:

> Hi,
>
> We're struggling to get our build pipeline working on CircleCI, which has a
> 4GB RAM limit. Here are some project stats to set the context:
>
> - 1,200+ modules
> - 36,315 LoC of Haskell
> - On the local machine with -O1 -j the build takes approx 5.2 GB of RAM [1]
>
> Trying to build on CircleCI with -O1 -j fails consistently, as expected.
> However, the behaviour of -M -j -O1 is erratic, at best. Sometimes it
> magically works, sometimes it fails. A number of times we've seen GHC take
> 3.5 GB (as reported by `top`), even though the limit was set to 2.5GB. Here
> are flags that we've tried in various combinations we've tried. None of
> them is consistent in building / failing:
>
> * -O1
> * -j
> * +RTS -M262144 -RTS
> * +RTS -A32m -RTS
> * +RTS -n2m -RTS
>
> What is the **real** behaviour of the -M option? How does it interact with
> -j and -A?
>
Did you ever make any progress on this, Saurabh?

The summary is this:

 * -j just tells GHC to parallelise compilation across modules. This can
increase the maximum heap size needed by the compiler.

 * -A sets the nursery size; to first order the doesn't affect the
maximum heap size, but rather is helpful when running parallel
programs (e.g. ghc with -j) to minimize the frequency with which we
must garbage-collect.

 * -M is a bit tricky to define. For one, it defines the maximum heap
   size beyond which we will terminate. However, we also use it in
   garbage collector to make various decisions about GC scheduling. I'll
   admit that I'm not terribly familiar with the details here.

Note that -M does not guarantee that GHC will find a way to keep your
program under the limit that you provide. It merely ensures that the
program doesn't exceed the given size, aborting if necessary.

At least this is my understanding.

Cheers,

- Ben



signature.asc
Description: PGP signature
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs