On Thu, May 17, 2007 at 03:46:59PM +0200, Hannes Dorbath wrote: > On a clone box I disabled overcommit, lowered PG's memory settings a > bit, disabled swap, mirrored production load to it and monitored it how > it would behave. As I more or less expected, it got into trouble after > about 6 hours. All memory was exhausted, it was even unable to fork bash > again. To my surprise I haven't found any evidence of OOM going active > in the logs.
I think you are misunderstanding what overcommit does. Normally when you're running programs and they fork(), the memory gets marked copy-on-write. This data is only once in memory but if it is written by one of the programs, they get they're own copy. Thus, it's memory allocated but not actually used. Thus "overcommit". Normally this is never a problem, but say that some unusual load happens and every process with shared usage actually wants it' own copy, and there's not enough memory+swap to hold it, something has to give. Thus the OOM killer. By disabling overcommit, all that happens is that in the above situation, if the kernel sees it's overcomitting total memry+swap, it returns ENOMEM instead. Thus instead of the unpredicatable OOM failure, you get unpredicatable fork()/malloc()/exec() failure. For example you can't start any more processes. > I blamed this behaviour to the swap space I've taken away, and not to > disabling overcommit. However I just enabled overcommit again and tried > to reproduce the behaviour. I was unable to get it into trouble again, > even with artificial high load. The default setting under Linux with overcommit off are that the total "allocated" pages in the system cannot exceed swap + 50% of memory. Thus by removing the swap you severely limited the amount of memory that could be used by programs. You need to give at least as much swap as memory, otherwise you'll never get the most out of your machine. > 1.) Why does it behave different when only changing overcommit? To my > understanding it should have run out of memory in both cases, or can PG > benefit from enabled overcommit? It's a minimal setup with PG being the > only one using any noticeable amount of resources. I hope I've answered you question above. Personally I don't disable overcommit as I find OOM less irritating that not being able to login when the machine is in trouble. > 2.) Is it possible at all to put a cap on the memory PG uses in total > from the OS side? kernel.shmmax, etc only limit some type of how PG > might use memory? Of cause excluding OS/FS buffers etc. You can make limit per process (see ulimit) and limit the number of connections. If a postgres process runs out of memory, it aborts the query. > 3.) Can PG be made to use it's own temp files when it runs out of memory > without setting memory settings so low that performance for typical load > will be worse? I think it would be nice, if I wouldn't need s lot of > swap, just to be safe under any load. Shouldn't that be more efficient > than using paged out memory anyway? Nope, letting the OS page is far more efficient than anything postgres can do. > Currently it seems to me that I have to sacrifice the performance of > typical load, when disabling overcommit and / or reducing swap, as I > have to push PG's memory settings lower to be safe. Make lots and lots of swap. You'll probably never use it, but at least it won't get in your way. I'd say 1-1.5 times your memory at least if you want overcommit off. Have a nice day, -- Martijn van Oosterhout <[EMAIL PROTECTED]> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to > litigate.
signature.asc
Description: Digital signature
