On Fri, 6 Mar 2020 16:06:00 +0800
hero...@gentoo.org wrote:
> From: Benda Xu
>
> If ghc spawns too many C compilers, it will exhaust file descripters.
I don't think ghc spawns more than 1 parallel gcc per compiled haskell file.
I'd expect a small constant overhead of file descriptors per compilation thread,
something like 5 fds (not 16 as your '64' limit implies):
- 2 opened files for write a result (one .hi and one .o file)
- 3 file descriptors per external tool to handle pipe std{in,out,err}.
Unless it's a -split-objs effect (which I don't believe is done in parallel),
then
the FD limit is orthogonal to job count.
Running 'strace -f -y' against `ghc --make` invocation should make it obvious.
Please file a bug to find out where these file descriptors come from.
Once we understand better where descriptors come from the bug might
need to go upstream eventually to more explicitly manage scarce fd resources
on the side of compilation manager. There should be no reason to limit
parallelism
as long as there is no external process forking.
> In the reference, it was thought to be a macOS bug for aggressive fd
> limits. But the ghc bug also applies to GNU/Linux, when ghc is
> asked to spawn, for example 256, jobs.
>
> This patch circumvents this ghc design flaw.
I don't see it as a design flaw.
> Reference: https://github.com/commercialhaskell/stack/issues/1177
> Reference: https://github.com/commercialhaskell/stack/issues/1979
> Signed-off-by: Benda Xu
> ---
> eclass/ghc-package.eclass | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/eclass/ghc-package.eclass b/eclass/ghc-package.eclass
> index 5361f09af1e9..d729f4a407b4 100644
> --- a/eclass/ghc-package.eclass
> +++ b/eclass/ghc-package.eclass
> @@ -203,7 +203,9 @@ ghc-make-args() {
> #https://ghc.haskell.org/trac/ghc/ticket/9221#comment:57
> # SMP is a requirement for parallel GC's gen0
> # 'qb' balancing.
> - echo "-j$(makeopts_jobs) +RTS -A256M -qb0 -RTS"
> + local n=$(makeopts_jobs)
> + [[ ${n} -gt 64 ]] && n=64
> + echo "-j${n} +RTS -A256M -qb0 -RTS"
Needs an in-source comment why a limit is imposed. Otherwise looks ok to push
while we are figuring out the limits. And a an explanation how it was
calculated.
> ghc_make_args=()
> fi
> echo "${ghc_make_args[@]}"
> --
> 2.25.0
>
>
--
Sergei
