On May 4, 2016, at 9:49 PM, Hervé Pagès <hpa...@fredhutch.org> wrote:
> Hi,
>
> Interesting "feature" in 10.11.4. I wonder if the process is killed
> before or after malloc() returns. If before, it seems very blunt:
> "You're asking too much and I don't like it so I kill you now".
> If after it doesn't look much better: "You're asking a lot and I
> don't like it but I give it to you anyway. I'll kill you quickly
> later".
>
> Why wouldn't the kernel just refuse to give that memory instead
> (i.e. malloc() returns NULL).
>
> Just curious...
>
You can google it, but this seems to be standard practice in modern OSes
(including OS X and Linux) where the kernel is not actually allocating the
physical memory when the application asks for it so malloc() will succeed and
give only a pointer to a virtual space where it will be mapping physical memory
into as needed (aka opportunistic allocation). However, when the OS figures out
later that it is overcommitted, it has no good way to signal the app that it
can't use what it was promised, so it will use SIGKILL. The rationale is that
in most cases the kernel can scramble to fulfill the request so it gives it
more flexibility in managing the memory (it can spread pages to RAM, swap,
shared ,...), but when it cannot recover it has no choice but to kill the
process.
I don't have the debugger open anymore, but if I recall correctly in my case
the kill happened in copyVector() -> duplicate(), so it was not necessarily on
the initial allocation but rather further down the line.
Cheers,
Simon
> H.
>
> On 05/04/2016 06:21 PM, Simon Urbanek wrote:
>>
>> On May 4, 2016, at 9:00 PM, Marius Hofert <marius.hof...@uwaterloo.ca> wrote:
>>
>>> Hi Simon,
>>>
>>> thanks for your quick reply.
>>>
>>> 1) ... so you can reproduce this?
>>
>> Yes, I can on 10.11.4.
>>
>>
>>> 2) Do you know a way how this can be 'foreseen'? We allocate larger
>>> matrices in the copula package depending on the user's input
>>> dimension. It would be good to tell her/him "Your dimension is quite
>>> large. Be aware of killers in your neighborhood"... before the killer
>>> attacks.
>>>
>>
>> Not directly, because the system is happy to accommodate R's request for
>> more memory but then it will strike out of the blue. Since that decision is
>> made by the kernel and it doesn't expose its thinking about how it feels,
>> it's hard to tell. I couldn't find any API on OS X akin to
>> didReceiveMemoryWarning in iOS, unfortunately.
>>
>> You could guess - e.g. by checking the total system memory
>>
>> int mib [] = { CTL_HW, HW_MEMSIZE };
>> int64_t value = 0;
>> size_t length = sizeof(value);
>> sysctl(mib, 2, &value, &length, NULL, 0);
>>
>> and comparing it to the sizes involved. However, even that is not foolproof,
>> because it all depends on the other processes' memory usage, swap space size
>> etc. There are slightly more involved ways to query the VM system as well,
>> but I'm not sure I'd want to go so deep into the weeds, especially since
>> this becomes quickly highly OS-specific.
>>
>> Cheers,
>> Simon
>>
>>
>>
>>> Thanks & cheers,
>>> Marius
>>>
>>>
>>> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
>>> <simon.urba...@r-project.org> wrote:
>>>>
>>>> On May 4, 2016, at 6:14 PM, Marius Hofert <marius.hof...@uwaterloo.ca>
>>>> wrote:
>>>>
>>>>>> Can you elaborate on "leads to R being killed"? You should tell to the
>>>>>> killer not to do it again :).
>>>>>
>>>>> Hi Simon!
>>>>>
>>>>> Sure, but who do you tell it if you don't know the killer?
>>>>> This is all the killer left me with, the 'crime scene' if you like :-)
>>>>>
>>>>>> m <- matrix(0, 90000, 100000)
>>>>> Killed: 9
>>>>>
>>>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>>>> apparently the killer went further down the hallway to his office
>>>>> now... so scary. Here is Wayne's sessionInfo():
>>>>>
>>>>
>>>> Yes, indeed, scary - since it means someone is killing R which means there
>>>> is not much R itself can do about it. In fact from the syslog I see
>>>>
>>>> May 4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>>>>
>>>> so it's the kernel's own defense mechanism. The bad thing is that R cannot
>>>> do anything about it - the kernel just decides to snipe processes it
>>>> thinks are dangerous to the health of the system, and it does so without a
>>>> warning.
>>>>
>>>> Cheers,
>>>> Simon
>>>>
>>>
>>
>> ______________________________________________
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpa...@fredhutch.org
> Phone: (206) 667-5791
> Fax: (206) 667-1319
>
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
