[PATCH] Attempt to increase RLIMIT_STACK in the driver as well as compiler (PR c++/49756, take 2)

[Jakub Jelinek]

On Mon, Jul 18, 2011 at 11:58:38PM +0200, Jakub Jelinek wrote:
> Especially the FEs and gimplification are highly recursive on more complex
> (especially badly generated) testcases, the following patch attempts to
> increase stack size to 64MB if possible.  It is done in the driver
> (where it can affect the children of the driver early) and also in
> toplev_main to make similar results when invoking the compiler by hand,
> unless mmap of some shared library or some other mmap make it impossible
> to have such a large stack.
> 
> Bootstrapped/regtested on x86_64-linux and i686-linux, cures
> gcc.c-torture/compile/limits-exprparen.c ICEs on x86_64-linux on all
> -O* levels as well as the testcase from this PR (which is quite large and
> compile time consuming, so not including it in this testcase).

Here is a slightly modified variant which will avoid the setrlimit syscall
if it wouldn't change anything (which is either if the soft limit is already
>= 64MB or the hard limit is not infinity and equal to the soft limit).
At least in toplev_main it is very likely the setrlimit will not be needed
if it has been done in the driver already.

2011-07-19  Jakub Jelinek  <ja...@redhat.com>

        PR c++/49756
        * gcc.c (main): Try to increase RLIMIT_STACK to at least
        64MB if possible.
        * toplev.c (toplev_main): Likewise.

--- gcc/gcc.c.jj        2011-07-08 15:09:38.000000000 +0200
+++ gcc/gcc.c   2011-07-19 10:15:25.000000000 +0200
@@ -6156,6 +6156,24 @@ main (int argc, char **argv)
   signal (SIGCHLD, SIG_DFL);
 #endif
 
+#if defined(HAVE_SETRLIMIT) && defined(RLIMIT_STACK) && defined(RLIM_INFINITY)
+  {
+    /* Parsing and gimplification sometimes need quite large stack.
+       Increase stack size limits if possible.  */
+    struct rlimit rlim;
+    if (getrlimit (RLIMIT_STACK, &rlim) == 0
+       && rlim.rlim_cur != RLIM_INFINITY
+       && rlim.rlim_cur < 64 * 1024 * 1024
+       && (rlim.rlim_max == RLIM_INFINITY || rlim.rlim_cur < rlim.rlim_max))
+      {
+       rlim.rlim_cur = 64 * 1024 * 1024;
+       if (rlim.rlim_max != RLIM_INFINITY)
+         rlim.rlim_cur = MIN (rlim.rlim_cur, rlim.rlim_max);
+       setrlimit (RLIMIT_STACK, &rlim);
+      }
+  }
+#endif
+
   /* Allocate the argument vector.  */
   alloc_args ();
 
--- gcc/toplev.c.jj     2011-07-11 10:39:50.000000000 +0200
+++ gcc/toplev.c        2011-07-19 10:15:37.000000000 +0200
@@ -1911,6 +1911,24 @@ do_compile (void)
 int
 toplev_main (int argc, char **argv)
 {
+#if defined(HAVE_SETRLIMIT) && defined(RLIMIT_STACK) && defined(RLIM_INFINITY)
+  {
+    /* Parsing and gimplification sometimes need quite large stack.
+       Increase stack size limits if possible.  */
+    struct rlimit rlim;
+    if (getrlimit (RLIMIT_STACK, &rlim) == 0
+       && rlim.rlim_cur != RLIM_INFINITY
+       && rlim.rlim_cur < 64 * 1024 * 1024
+       && (rlim.rlim_max == RLIM_INFINITY || rlim.rlim_cur < rlim.rlim_max))
+      {
+       rlim.rlim_cur = 64 * 1024 * 1024;
+       if (rlim.rlim_max != RLIM_INFINITY)
+         rlim.rlim_cur = MIN (rlim.rlim_cur, rlim.rlim_max);
+       setrlimit (RLIMIT_STACK, &rlim);
+      }
+  }
+#endif
+
   expandargv (&argc, &argv);
 
   /* Initialization of GCC's environment, and diagnostics.  */

        Jakub