Re: [HACKERS] tuning autovacuum

2011-10-14 Thread Bruce Momjian
Robert Haas wrote: On Wed, Jun 8, 2011 at 10:55 PM, Euler Taveira de Oliveira eu...@timbira.com wrote: Em 08-06-2011 20:35, Robert Haas escreveu: Is the hint correct? ?I mean, what if there were 100 small tables that needed vacuuming all at the same time. ?We'd hit this limit no matter

Re: [HACKERS] tuning autovacuum

2011-10-14 Thread Josh Berkus
Ideally we would have something like checkpoint_warning that warns users in the log when there are too few autovacuum workers and cleanup is being delayed. I don't think that any table-stats based approach is going to work. I think you need to measure the queue of tables which need

Re: [HACKERS] tuning autovacuum

2011-10-14 Thread Robert Haas
On Fri, Oct 14, 2011 at 12:59 PM, Josh Berkus j...@agliodbs.com wrote: Ideally we would have something like checkpoint_warning that warns users in the log when there are too few autovacuum workers and cleanup is being delayed. I don't think that any table-stats based approach is going to

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Greg Smith
Robert Haas wrote: Well, if there are more tables that need vacuuming than there are workers available at any given time, there will be a delay. We probably don't keep track of that delay at present, but we could. There are at least four interesting numbers to collect each time autovacuum

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Alvaro Herrera
Excerpts from Tom Lane's message of mié jun 08 21:50:22 -0400 2011: Robert Haas robertmh...@gmail.com writes: I think it'd be really useful to expose some more data in this area though. One random idea is - remember the time at which a table was first observed to need vacuuming. Clear the

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: In any case, given the rebalancing feature of vacuum_cost_delay (which increases the delay the more workers there are), the only solution to the problem of falling behind is reducing the delay parameter. If you just add more workers, they

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Robert Haas
On Thu, Jun 9, 2011 at 12:25 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Excerpts from Tom Lane's message of mié jun 08 21:50:22 -0400 2011: Robert Haas robertmh...@gmail.com writes: I think it'd be really useful to expose some more data in this area though.  One random idea is -

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Bernd Helmle
--On 9. Juni 2011 11:53:22 -0400 Greg Smith g...@2ndquadrant.com wrote: There are at least four interesting numbers to collect each time autovacuum runs: 1) This one, when was the threshold crossed. I believe one of the AV workers would have to pause periodically to update these if they're

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Greg Smith
On 06/09/2011 04:43 PM, Bernd Helmle wrote: I'd go further and expose the info or details issued by VACUUM VERBOSE into the view, too, at least the number of pages visited and cleaned (or dead but not yet cleaned). Customers are heavily interested in these numbers and i've found pgfouine to

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes: What we'd probably need to do with those is handle them like the other stats in the system: store a total number for visited/cleaned/dead for each relation, then increment the total as each vacuum finishes. As Robert said, we're already seeing

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Greg Smith
On 06/09/2011 05:41 PM, Tom Lane wrote: As Robert said, we're already seeing scalability problems with the pg_stats subsystem. I'm not eager to add a bunch more per-table counters, at least not without some prior work to damp down the ensuing performance hit. That's fair. Anyone who is

Re: [HACKERS] tuning autovacuum

2011-06-09 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes: That's fair. Anyone who is running into the sort of autovacuum issues prompting this discussion would happily pay the overhead to get better management of that; it's one of the easiest things to justify more per-table stats on IMHO. Surely the

Re: [HACKERS] tuning autovacuum

2011-06-08 Thread Magnus Hagander
On Jun 9, 2011 12:01 AM, Euler Taveira de Oliveira eu...@timbira.com wrote: Hi, There are some releases that autovacuum was enabled by default and, up to now there is an easy way to estimate the number of autovacuum workers. I tune it observing if the number of slots are saturated for a period

Re: [HACKERS] tuning autovacuum

2011-06-08 Thread Robert Haas
On Wed, Jun 8, 2011 at 5:54 PM, Euler Taveira de Oliveira eu...@timbira.com wrote: LOG: maximum number of autovacuum workers reached HINT: Consider increasing autovacuum_max_workers (currently 5). Comments? Is the hint correct? I mean, what if there were 100 small tables that needed

Re: [HACKERS] tuning autovacuum

2011-06-08 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I think it'd be really useful to expose some more data in this area though. One random idea is - remember the time at which a table was first observed to need vacuuming. Clear the timestamp when it gets vacuumed. Then you can do: As far as I recall

Re: [HACKERS] tuning autovacuum

2011-06-08 Thread Robert Haas
On Wed, Jun 8, 2011 at 9:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I think it'd be really useful to expose some more data in this area though.  One random idea is - remember the time at which a table was first observed to need vacuuming. Clear the

Re: [HACKERS] tuning autovacuum

2011-06-08 Thread Euler Taveira de Oliveira
Em 08-06-2011 20:35, Robert Haas escreveu: Is the hint correct? I mean, what if there were 100 small tables that needed vacuuming all at the same time. We'd hit this limit no matter how high you set autovacuum_max_workers, but it wouldn't be right to set it to 101 just because every once in a

Re: [HACKERS] tuning autovacuum

2011-06-08 Thread Robert Haas
On Wed, Jun 8, 2011 at 10:55 PM, Euler Taveira de Oliveira eu...@timbira.com wrote: Em 08-06-2011 20:35, Robert Haas escreveu: Is the hint correct?  I mean, what if there were 100 small tables that needed vacuuming all at the same time.  We'd hit this limit no matter how high you set