Re: fork/exec ... unable to allocate memory

2015-06-03 Thread Menno Smits
This thread and the ticket linked by Michael got me curious about whether we could write our own routine for spawning processes that doesn't invoke the usual copy-on-write semantics. The struct returned by exec.Command has a a SysProcAttr field where you can set (Linux-specific) flags to pass to

Re: fork/exec ... unable to allocate memory

2015-06-03 Thread Menno Smits
On 4 June 2015 at 11:56, Menno Smits menno.sm...@canonical.com wrote: bigs := make([][]byte, 6) Note: I was using 6GB because my machine is running a bunch of VMs and has very little free memory at the moment. With the 14GB allocated in the example, memory runs out before the program gets

Re: fork/exec ... unable to allocate memory

2015-06-03 Thread Michael Hudson-Doyle
If it is overcommit, this go issue will make for interesting (but possibly not helpful) reading: https://github.com/golang/go/issues/5838 On 3 June 2015 at 17:24, Gustavo Niemeyer gust...@niemeyer.net wrote: Hey John, It's probably an overcommit issue. Even if you don't have the memory in use,

Re: fork/exec ... unable to allocate memory

2015-06-03 Thread John Meinel
So interestingly we are already fairly heavily overcommitted. We have 4GB of RAM and 4GB of swap available. And cat /proc/meminfo is saying: CommitLimit: 6214344 kB Committed_AS:9764580 kB John =:- On Wed, Jun 3, 2015 at 9:28 AM, Gustavo Niemeyer gust...@niemeyer.net wrote: Ah, and

Re: fork/exec ... unable to allocate memory

2015-06-03 Thread John Meinel
Yeah, I'm pretty sure this machine is on 0 and we've just overcommitted enough that Linux is refusing to overcommit more. I'm pretty sure juju was at least at 2GB of pages, where 1G was in RAM and 1GB was in swap. And if we've already overcommitted to 9.7GB over 6.2GB linux probably decided that

Re: fork/exec ... unable to allocate memory

2015-06-03 Thread Gustavo Niemeyer
From https://www.kernel.org/doc/Documentation/vm/overcommit-accounting: The Linux kernel supports the following overcommit handling modes 0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It

fork/exec ... unable to allocate memory

2015-06-02 Thread John Meinel
So we're running into this failure mode again at one of our sites. Specifically, the system is running with a reasonable number of nodes (~100) and has been running for a while. It appears that it wanted to restart itself (I don't think it restarted jujud, but I do think it at least restarted a

Re: fork/exec ... unable to allocate memory

2015-06-02 Thread Gustavo Niemeyer
Hey John, It's probably an overcommit issue. Even if you don't have the memory in use, cloning it would mean the new process would have a chance to change that memory and thus require real memory pages, which the system obviously cannot give it. You can workaround that by explicitly enabling

Re: fork/exec ... unable to allocate memory

2015-06-02 Thread Gustavo Niemeyer
Ah, and you can also suggest increasing the swap. It would not actually be used, but the system would be able to commit to the amount of memory required, if it really had to. On Jun 3, 2015 1:24 AM, Gustavo Niemeyer gust...@niemeyer.net wrote: Hey John, It's probably an overcommit issue. Even