[HACKERS] Limit allocated memory per session

2009-10-01 Thread daveg
I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. The problem this addresses is that sometimes one needs to set work_mem fairly high to get good query plans for large joins. However, some complex queries will then use huge amounts of memory

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Tom Lane
daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Alvaro Herrera
daveg wrote: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. The problem this addresses is that sometimes one needs to set work_mem fairly high to get good query plans for large joins. However, some complex queries will then use huge

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Euler Taveira de Oliveira
Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have ulimit? -- Euler Taveira de Oliveira http://www.timbira.com/ --

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Tom Lane
Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have ulimit? Get a real

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Bill Moran
In response to Euler Taveira de Oliveira eu...@timbira.com: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. What about plataforms (Windows) that don't have

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Robert Haas
On Thu, Oct 1, 2009 at 11:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit.

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: What WOULD be useful is to find a way to provide a way to configure work_mem per backend rather than per executor node. But that's a much harder problem. I think it's mostly a planner problem: how do you deal with the fact that that would make cost

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Euler Taveira de Oliveira
Robert Haas escreveu: On Thu, Oct 1, 2009 at 11:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Tom Lane
Euler Taveira de Oliveira eu...@timbira.com writes: I see. Tough problem is: how do we get per backend memory usage accurately? Is it relying on OS specific API the only way? Given all the third-party libraries (perl, python, libxml2, yadda yadda) that can be in use and won't go through palloc,

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Euler Taveira de Oliveira
Tom Lane escreveu: Robert Haas robertmh...@gmail.com writes: What WOULD be useful is to find a way to provide a way to configure work_mem per backend rather than per executor node. But that's a much harder problem. I think it's mostly a planner problem: how do you deal with the fact that

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Oh, BTW, did anyone get interested in adding the bits to disable the OOM killer for postmaster on the various Linux initscripts? It needs some games with /proc/pid/oom_adj and requires root privileges, but I think an initscript is in an

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: I was imagining that this would be something for individual distros to tackle. It's probably not portable enough to go into the contrib/start-scripts examples. On the other hand, it'd make lots of sense to have the Fedora or Debian or whatever scripts

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Alvaro Herrera
Tom Lane wrote: Alvaro Herrera alvhe...@commandprompt.com writes: Oh, BTW, did anyone get interested in adding the bits to disable the OOM killer for postmaster on the various Linux initscripts? It needs some games with /proc/pid/oom_adj and requires root privileges, but I think an

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: I'm not exactly keen on Debian init scripts hacking kernel settings. Should it hack up the shared memory numbers too? This is not what I would consider 'init script' material for specific applications. What was suggested was tweaking the oom_adj

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread daveg
On Thu, Oct 01, 2009 at 10:35:55AM -0400, Tom Lane wrote: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit. That was my initial thought too. However, ulimit() is documented as superceded by

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread daveg
On Thu, Oct 01, 2009 at 11:47:43AM -0400, Tom Lane wrote: Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to propose adding a new GUC to limit the amount of memory a backend can allocate for its own use. Use ulimit.

Re: [HACKERS] Limit allocated memory per session

2009-10-01 Thread Robert Haas
On Thu, Oct 1, 2009 at 12:15 PM, Euler Taveira de Oliveira eu...@timbira.com wrote: Robert Haas escreveu: On Thu, Oct 1, 2009 at 11:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: Euler Taveira de Oliveira eu...@timbira.com writes: Tom Lane escreveu: daveg da...@sonic.net writes: I'd like to