This makes my head spin. ridiculous that an over commit even exists. any slight performance gain is totally nullified by a crashing program.
--------------------- Scott Doctor scott at scottdoctor.com --------------------- On 4/30/2015 11:47 AM, Scott Robison wrote: > On Thu, Apr 30, 2015 at 11:42 AM, James K. Lowden <jklowden at > schemamania.org> > wrote: > >> On Wed, 29 Apr 2015 20:29:07 -0600 >> Scott Robison <scott at casaderobison.com> wrote: >> >>>> That code can fail on a system configured to overcommit memory. By >>>> that standard, the pointer is invalid. >>>> >>> Accidentally sent before I was finished. In any case, by "invalid >>> pointer" I did not mean to imply "it returns a bit pattern that could >>> never represent a valid pointer". I mean "if you dereference a >>> pointer returned by malloc that is not null or some implementation >>> defined value, it should not result in an invalid memory access". >> Agreed. And I don't think that will happen with malloc. It might, and >> I have a plausible scenario, but I don't think that's what happened. >> > The Linux man page for malloc documents that the pointer returned may not > be usable in the case of optimistic memory allocations, as the eventual use > of the pointer may trigger the need to commit a page of memory to the > address space and that a page of memory may not be available at that point > in time. Thus malloc, on Linux, makes no guarantee as to the viability of > using the returned pointer. > > Perhaps you are correct and "sigsegv" is not the literal signal that is > triggered in this case. I don't care, really. The fact is that an > apparently valid pointer was returned from a memory allocation function yet > can result in an invalid access for whatever reason (out of memory, in this > case). The Linux OOM killer may kill the offending process (which is what > one would expect, but one would also expect malloc to return null, so we > already know not to expect the expected). Or it may kill some other process > which has done nothing wrong! Sure, the OS is protecting the two processes > address space from one another, but it seems to me that if one process can > kill another process, there is a problem. > > I can see the utility of a memory allocation strategy like this. It should > in no way be the *default* memory allocation strategy, especially for a > system that tries to be posix compliant, because this is in direct > violation of posix compliance (by my reading) from > http://pubs.opengroup.org/onlinepubs/009695399/functions/malloc.html > > Upon successful completion with *size* not equal to 0, *malloc*() shall >> return a pointer to the allocated space. >> > Or maybe posix just needs a better definition for "allocated space". Sure, > an address was allocated in the processes address space, but actual memory > was not allocated. > > The decades old interface contract was "if you call malloc with a non-zero > size, you can depend on malloc to return a null pointer or a pointer to the > first byte of an uninitialized allocation". Thus your application could > decide what to do if the memory was not available: abort, exit, select an > alternative code path that can get the job done with less or no memory, > ignore the return value and let sigsegv handle it later. > > Now with optimistic memory allocation, you do not have a choice. If your > malloc call results in an overcommit, your process can be killed later when > it tries to access the memory. Or some other innocent process might be > killed. > > I really wonder how many man hours have been wasted trying to debug > problems with processes just to find out that the killed process did > nothing wrong, it was some other process overcommitting memory. Or worse, > how many man hours were wasted and no good reason was ever learned. > > I came across this last night while learning more about OOM: > https://lwn.net/Articles/104179/ -- particular, the analogy, which I think > is spot on. > > I realize that there is no one right answer to how an OS should handle > memory exhaustion. There are various tradeoffs. However, C is not an > operating system, it is a language, and the standards tell you how you can > expect it to behave. In this case, the C API is broken on Linux by default, > so it becomes impossible to write fault tolerant applications in the face > of this feature. >
