[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Paul D. Smith]

Update of bug #18396 (project make):

  Status:None => Fixed  
 Assigned to:None => psmith 
 Open/Closed:Open => Closed 
   Fixed Release:None => 3.82   

___

Follow-up Comment #7:

Starting with GNU make 3.82, we reset the stack limit back to its original
value before we exec the child.

Also note that if you are using posix_spawn() instead of fork/exec, we will
never reset the stack limit since posix_spawn() doesn't provide a way to reset
it.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Gabor Melis]

Follow-up Comment #6, bug #18396 (project make):

I've been recently caught out by setrlimit. I believe what happened is that
pthread_create (in a child process) stack size default is on linux calculated
from the stack limit. 


From the pthread_create man page:

   On Linux/x86-32, the default stack size for a new thread is 2
megabytes.
   Under the NPTL threading implementation, if the RLIMIT_STACK soft
resource
   limit at the time the program started has any value other than
unlimited,
   then it determines the default stack size of new threads.

If we start a program that calls pthread_create from the shell that has a soft
limit of 8192k and no hard limit, then the new thread will get a 8192k stack.
If the same program is started from make, then it will get a stack of 2048k
because make removes the soft limit.

I understand that calling pthread_create without setting the stack size is not
recommended and that the defaulting mechanism is suspect. Still, it's
surprising to find that make mucks with the limits.

A way out could be to actually restore the limits before the child is exec'ed
although that may not be legal or run into problems if the stack is in fact
deeper than the original limit.

___

Reply to this item at:

  http://savannah.gnu.org/bugs/?18396

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[anonymous]

Follow-up Comment #5, bug #18396 (project make):

I think this is not an enhancement request but rather a bug-fix request.

___

Reply to this item at:

  http://savannah.gnu.org/bugs/?18396

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[anonymous]

Follow-up Comment #4, bug #18396 (project make):

I experienced an converse effect.  I could run a program (kpdf) from shell,
but it would crash if started from make.

I used a large hard limit (~1GB) for the stack size and around 100MB as soft
limit (used it for a scientific app).  My machine has 2 GB Ram.

kpdf fails to start a helper thread with this message:

  QThread::start: thread creation error: Cannot allocate memory

The problem seems to be that make ups the soft limit to the hard limit and
runs its childs with this setting.

One can certainly argue, that kpdf behaves disadvantageous, however, make
should not change the user's specified environment.

___

Reply to this item at:

  http://savannah.gnu.org/bugs/?18396

___
  Nachricht geschickt von/durch Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Martin Dorey]

 However, your point about programs invoked by make inheriting the 
 setrlimit() is definitely something that seems problematic.  Perhaps
GNU 
 make could change the stack limit back to what it was after it forks
but 
 before it execs its child.  I wonder what happens if you change a
limit to 
 something too small for the current processes resources?

It doesn't look specified by
http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html.

This test suggests that it works fine on my Linux box.  I presume the
limit check is normally done on a faulting access off the end of the
stack, so setrlimit would need some to have some different and special
code to check the current stack usage if it wanted to apply the check
immediately.  It seems a bit unlikely that anyone would have gone to
that effort, thus stopping you from doing something useful in this sort
of situation.

#include alloca.h
#include errno.h
#include iostream
#include stddef.h
#include string.h
#include sys/time.h
#include sys/resource.h

void useLotsOfStack(size_t size);

size_t getAndDisplayStackLimit() {
  rlimit resourceLimit;
  if (getrlimit(RLIMIT_STACK, resourceLimit)  0) {
std::cerr  getrlimit failed with   errno  std::endl;
  }
  std::cout  rlim_cur ==   resourceLimit.rlim_cur  std::endl;
  std::cout  rlim_max ==   resourceLimit.rlim_max  std::endl;
  return resourceLimit.rlim_cur;
}

void setStackLimit(size_t size) {
  rlimit resourceLimit;
  resourceLimit.rlim_cur = size;
  if (setrlimit(RLIMIT_STACK, resourceLimit)  0) {
std::cerr  setrlimit failed with   errno  std::endl;
exit(1);
  }
  getAndDisplayStackLimit();
}

int main() {
  size_t initialLimit = getAndDisplayStackLimit();
  //useLotsOfStack(initialLimit * 10);
  setStackLimit(initialLimit * 100);
  useLotsOfStack(initialLimit * 10);
  setStackLimit(initialLimit);
  useLotsOfStack(initialLimit * 10);
}

void useLotsOfStack(size_t size) {
  memset(alloca(size), 0, size);
}
-
Martin's Outlook, BlueArc Engineering


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Howard Chu]

Martin Dorey wrote:
However, your point about programs invoked by make inheriting the 
setrlimit() is definitely something that seems problematic.  Perhaps

GNU 
  

make could change the stack limit back to what it was after it forks

but 
  

before it execs its child.  I wonder what happens if you change a

limit to 
  

something too small for the current processes resources?



It doesn't look specified by
http://www.opengroup.org/onlinepubs/009695399/functions/getrlimit.html.

This test suggests that it works fine on my Linux box.  I presume the
limit check is normally done on a faulting access off the end of the
stack, so setrlimit would need some to have some different and special
code to check the current stack usage if it wanted to apply the check
immediately.  It seems a bit unlikely that anyone would have gone to
that effort, thus stopping you from doing something useful in this sort
of situation.
  


In Linux (and most other Unix systems) the stack is allocated in 
page-sized chunks, and an extra page with no access permissions is 
mapped at the end of it. Accessing that page is what generates the SEGV 
that occurs when you overrun the stack. The mappings are only 
established when the stack is grown, and generally the stack never 
shrinks. So if you already have X pages of stack legitimately in use, 
lowering the rlimit isn't going to have an immediate effect. I.e., the 
stack growth occurred before the limit decreased. In general, Unix 
doesn't impose restrictions retroactively. (E.g., if you open a file and 
have a valid descriptor on it, and someone else chmods the file, 
removing your access permissions, you still have your original 
privileges through the open descriptor.)


--
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sunhttp://highlandsun.com/hyc
 OpenLDAP Core Teamhttp://www.openldap.org/project/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Jon Grant]

Hi,

Paul D. Smith elucidated on 29/11/06 02:27:
[...]
 Finally, there is no way to detect an out of stack error and exit gracefully
 with a warning as you suggest: the behavior of alloca() is undefined if you
 run out of stack space (it doesn't just return NULL as malloc() etc. do).

Is it undefined in actuality though? Has anyone checked if NULL does get
returned from any implementations? (I'm surprised alloca()
implementations didn't end up getting NULL returned in implementations
over the years. The GLIBC Manual indicates a fatal signal is generated
from its implementation:

http://www.gnu.org/software/libc/manual/html_node/Disadvantages-of-Alloca.html

My view would be that on modern computers switching to allocate from the
heap wouldn't make a big difference if it were changed. Modern heaps
have pools for small allocations to stop them fragmenting larger
allocations anyway. Someone would need to do a compressive test to know
for sure, these things often have knock on effects.. I've seen massive
slowdowns when someone switched malloc() to calloc() on MS-Windows!

Jon


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Howard Chu]

Jon Grant wrote:

My view would be that on modern computers switching to allocate from the
heap wouldn't make a big difference if it were changed. Modern heaps
have pools for small allocations to stop them fragmenting larger
allocations anyway. Someone would need to do a compressive test to know
for sure, these things often have knock on effects.. I've seen massive
slowdowns when someone switched malloc() to calloc() on MS-Windows!

Jon

  
Choice of malloc implementation can have a huge effect on execution 
time. See this data

http://highlandsun.com/hyc/#Malloc

Some modern mallocs are good, but stack-based allocation is still better 
a lot of the time. Especially for temporary variables that are just 
going to be discarded after a few computations.


--
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sunhttp://highlandsun.com/hyc
 OpenLDAP Core Teamhttp://www.openldap.org/project/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

Re: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Paul Smith]

On Wed, 2006-11-29 at 13:22 +, Jon Grant wrote:

  Finally, there is no way to detect an out of stack error and exit gracefully
  with a warning as you suggest: the behavior of alloca() is undefined if you
  run out of stack space (it doesn't just return NULL as malloc() etc. do).

 Is it undefined in actuality though? Has anyone checked if NULL does get
 returned from any implementations?

From the standpoint of GNU make it doesn't matter all that much what
some implementations do.  In order to be portable it must assume the
lowest common denominator.

 My view would be that on modern computers switching to allocate from the
 heap wouldn't make a big difference if it were changed. Modern heaps
 have pools for small allocations to stop them fragmenting larger
 allocations anyway. Someone would need to do a compressive test to know
 for sure, these things often have knock on effects.. I've seen massive
 slowdowns when someone switched malloc() to calloc() on MS-Windows!

Well, yes, but changing from malloc() to calloc() actually has a
significant difference: in the latter all the memory you've allocated
needs to be zeroed out.  That can have all kinds of implications even
beyond the obvious need to walk all the bytes in each allocation, in
terms of swapping, cache issues, etc.


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Scott McPeak]

Follow-up Comment #3, bug #18396 (project make):

Regarding efficiency:

First, the main point of my report is not alloca sucks but rather
that setrlimit is an unexpected thing for make to do.

But having opened up the can of worms, let me play with them a little
(I don't really want to start a big argument, but I do want to defend
my previous statements):

In my experience, the dominant effect on program performance, after
choice of algorithm, is data locality: frequently re-using the same or
nearby memory locations.  Instruction count is at best a second-order
effect.  Andrew Appel has a lovely paper where he proves that (of
all things) garbage collection is faster than stack allocation, by
exclusively focusing on instruction count:

  http://citeseer.ist.psu.edu/appel87garbage.html
  
Of course, the analysis is fatally flawed by ignoring locality.

My point regarding alloca is that if I have a lot of data, it cannot
all be hot (high locality).  If I put cold data on the stack, the
stack as a whole has poorer locality, because the hot regions (stack
frames) are interspersed with cold regions (the big chunks of data).
So if I intend to allocate infrequently-accessed data, it is better to
put it on the heap where it won't interfere with the locality of stack
accesses.

You say that only medium sized data ends up on the stack, which
certainly sounds like a reasonable approach, but if there is enough of
it to warrant calling setrlimit, then I have to suspect that some of
it would be better placed elsewhere, from a strictly
performance-oriented point of view.

Regarding portability:

Indeed, there is no portable way to detect running out of stack space,
whether by alloca or not, though alloca obviously makes running out of
stack space more likely.  Of course, setrlimit is not all that
portable either, so I was assuming that nonportable solutions were on
the table.  Toward that end, an appropriate signal handler (which will
need to use nonportable tricks to distinguish stack overflow from
other types of segfaults), or the GCC -fstack-check argument (when
compiling with GCC, which is after all the common case) may be options.

Moreover, I would argue that the architectural decision to make heavy
reliance on alloca is what leads us into nonportable territory in the
first place.  It is not uncommon to find systems with ~10MB or less of
stack space.  The setrlimit call is a band-aid; in some circumstances
it will help, but that call is not an antidote to running out of stack
space.

Of course, the big advantage of alloca is that it is easy to program
with.  I would not advocate suddenly rewriting GNU make to use malloc
everywhere, since (as you say) that will likely introduce many more
serious bugs than this one in the short term.  (If it was written in
C++ there would be better options, but that ship too has sailed.)

So, at most, I would suggest doing some profiling to find the biggest
consumers of alloca space and change just them (hopefully only a
handful) to use [x]malloc instead.  The primary motivation would be
enhanced portability; the efficiency argument is just meant to allay
fears that this will make everything slower.

Regarding setrlimit itself:

Yes, I would consider this bug (and I do consider it a bug...) to be
fixed if 'make' would reliably return the resource limits to their
original settings before exec'ing its children.


___

Reply to this item at:

  http://savannah.gnu.org/bugs/?18396

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Scott McPeak]

URL:
  http://savannah.gnu.org/bugs/?18396

 Summary: stack size setrlimit call interacts badly with
Solaris/x86 kernel bug
 Project: make
Submitted by: smcpeak
Submitted on: Tuesday 11/28/2006 at 20:16
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 3.81
Operating System: POSIX-Based
   Fixed Release: None

___

Details:

I recently experienced the following mysterious phenomenon:

* Running a program from the shell, it crashes (it dereferences NULL
on purpose; that is all it does).

* Invoking that same program from with a Makefile, it does not crash.
Instead, it happily reads/writes page 0.

After considerable investigation, I have discovered that the cause is
two interacting issues:

* There is a kernel bug in Solaris/x86 that causes page 0 to be mapped
if the stack limit is 'unlimited'; see

  http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6374692
  
* GNU make was modified on 2001-09-06 (between 3.79.1 and 3.80) by
Paul Eggert to Get rid of any avoidable limit on stack size (quote
from ChangeLog).  See the 'setrlimit' call in main.c.

Consequently, whether a program is running under 'make' greatly
affects how it behaves, as the child processes inherit the resource
limits as well.

While the kernel issue is clearly a bug, I think 'make' behavior is a
misfeature as well.  Generally one expects resource limits to not be
silently changed by shells and shell-like programs such as 'make'.
That 'make' does this is troubling; among other things, diagnosing the
consequences is difficult (I investigated many other possible causes
before finding it).  The Solaris kernel bug is just one way such a
silent change might be manifested.

If 'make' needs to allocate a large amount (megabytes) of data, it
would be better to do so on the heap, both from a portability
standpoint (the stack size) and from a performance standpoint (it
messes up the normally good locality of stack access).

Alternatively, if it must allocate on the stack, then detecting and
complaining about a too-low limit would be better in my opinion than
silently changing it.  It's easy to uncap the stack size explicitly
in build scripts and whatnot when truly necessary.

Output of uname -a:

  SunOS tainted.sf.coverity.com 5.10 Generic_118844-26 i86pc i386 i86pc
Solaris

Output of make -v:

  GNU Make 3.81
  Copyright (C) 2006  Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.
  There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
  PARTICULAR PURPOSE.

  This program built for i386-pc-solaris2.10





___

Reply to this item at:

  http://savannah.gnu.org/bugs/?18396

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Paul D. Smith]

Update of bug #18396 (project make):

  Item Group: Bug = Enhancement

___

Follow-up Comment #1:

 If 'make' needs to allocate a large amount (megabytes) of data,
 it would be better to do so on the heap, both from a
 portability standpoint (the stack size) and from a performance
 standpoint (it messes up the normally good locality of stack
 access).

 Alternatively, if it must allocate on the stack, then detecting
 and complaining about a too-low limit would be better in my
 opinion than silently changing it. It's easy to uncap the stack
 size explicitly in build scripts and whatnot when truly
 necessary.

Unfortunately, none of the above is true.  Make needs extra stack space
because it makes extensive use of the alloca() function.  It does not
allocate huge chunks of memory on the stack: if large amounts of memory are
needed they are allocated on the stack.  alloca() is used for modest-sized
memory chunks, to hold filenames and smaller strings (make is, at heart, a
string manipulation program).  Using heap, which requires a system call to
get more memory, for all of these small allocations would be much less
efficient than using the stack.  Not to mention the overhead of having to
allocate these memory segments to be used for a short time, then freeing them
again, over and over.

However, because make is also very recursive, even though no single alloca()
call is very large it's quite possible for the entirety of the alloca()
invocations for a complex makefile to use quite a bit of stack.

Of course, there's no way to determine how much stack will be used a priori,
since it depends entirely on the construction of the makefile, exactly which
functions are invoked in which order, and even the current state of the build
(whether various targets need to be rebuilt or not).  Further, there is no
portable way that I'm aware of to determine how much stack is left before the
program runs out.

Finally, there is no way to detect an out of stack error and exit gracefully
with a warning as you suggest: the behavior of alloca() is undefined if you
run out of stack space (it doesn't just return NULL as malloc() etc. do).

So.  In order to avoid the need for extra stack in GNU make all the
invocations of alloca() would need to be rewritten to use xmalloc(), and
those functions would need to be changed to be careful to free all that
memory whenever the function exited to avoid memory leaks... with what must
certainly be a decrease in performance (although of how much we can't be
sure).

On the other hand, as you point out, the problem on Solaris is a bug in the
kernel which you can hardly blame GNU make for.

However, your point about programs invoked by make inheriting the setrlimit()
is definitely something that seems problematic.  Perhaps GNU make could change
the stack limit back to what it was after it forks but before it execs its
child.  I wonder what happens if you change a limit to something too small
for the current processes resources?

___

Reply to this item at:

  http://savannah.gnu.org/bugs/?18396

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Paul D. Smith]

Follow-up Comment #2, bug #18396 (project make):

I wrote:

 if large amounts of memory are needed they are allocated on the stack

Of course I meant on the _heap_ :-/.

___

Reply to this item at:

  http://savannah.gnu.org/bugs/?18396

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Martin Dorey]

 Using heap, which requires a system call to get more memory

(It doesn't affect the main point of Paul's reply but just for academic
interest) no it doesn't:

[EMAIL PROTECTED]:~/playpen$ cat ten-thousand-mallocs.c 
#include stdlib.h

int main() {
  for (int ii = 0; ii != 10 * 1000; ++ ii) {
free(malloc(1));
  }
}
[EMAIL PROTECTED]:~/playpen$ g++ ten-thousand-mallocs.c 
[EMAIL PROTECTED]:~/playpen$ strace ./a.out 21 | wc -l
152
[EMAIL PROTECTED]:~/playpen$

Even in less contrived applications, brk isn't called anything like as
often as malloc.
-
Martin's Outlook, BlueArc Engineering


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make

RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug

[Paul Smith]

On Tue, 2006-11-28 at 18:45 -0800, Martin Dorey wrote:
  Using heap, which requires a system call to get more memory
 
 (It doesn't affect the main point of Paul's reply but just for academic
 interest) no it doesn't:

 Even in less contrived applications, brk isn't called anything like as
 often as malloc.

Certainly that's true... I didn't mean to suggest every call to malloc()
resulted in a system call.  But using the heap DOES require a system
call to get memory... obviously the C runtime will grab memory in much
larger chunks and maintains a free pool, with algorithms to handle
fragmentation and coalescing of free blocks, etc.  There are whole
dissertations written on the most efficient ways to manage heap :).

But (as I'm sure we both agree) alloca() is unquestionably more
efficient (assuming you have an appropriate amount of stack).


___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make