Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-12 Thread Alvaro Herrera
On Fri, Aug 12, 2005 at 03:49:57PM -0700, Mark Wong wrote: > On Fri, 12 Aug 2005 18:42:09 -0400 > Alvaro Herrera <[EMAIL PROTECTED]> wrote: > > Also, it seems the "tran_lock.out" file captured wrong input -- I think > > you mean "WHERE transactionid IS NULL" in the query instead of "WHERE > > tran

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-12 Thread Mark Wong
On Fri, 12 Aug 2005 18:42:09 -0400 Alvaro Herrera <[EMAIL PROTECTED]> wrote: > On Fri, Aug 12, 2005 at 03:16:04PM -0700, Mark Wong wrote: > > On Fri, 12 Aug 2005 17:49:41 -0400 > > Alvaro Herrera <[EMAIL PROTECTED]> wrote: > > > > > Notice how the subindexes are wrong ... I think it should be 1:3

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-12 Thread Alvaro Herrera
On Fri, Aug 12, 2005 at 03:16:04PM -0700, Mark Wong wrote: > On Fri, 12 Aug 2005 17:49:41 -0400 > Alvaro Herrera <[EMAIL PROTECTED]> wrote: > > > Notice how the subindexes are wrong ... I think it should be 1:3 for > > i_orders, no? Apparently indexes_scan.data has the same problem. > > Whoops!

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-12 Thread Alvaro Herrera
On Fri, Aug 12, 2005 at 10:49:43AM -0700, Mark Wong wrote: > I thought I'd run a couple of tests to see if it would be helpful > against CVS from Aug 3, 2005. > > Here's a run with autovacuum turned off: > http://www.testing.osdl.org/projects/dbt2dev/results/dev4-015/42/ > 5186.55 notpm > > Autva

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-12 Thread Mark Wong
On Fri, 12 Aug 2005 17:49:41 -0400 Alvaro Herrera <[EMAIL PROTECTED]> wrote: > On Fri, Aug 12, 2005 at 10:49:43AM -0700, Mark Wong wrote: > > I thought I'd run a couple of tests to see if it would be helpful > > against CVS from Aug 3, 2005. > > > > Here's a run with autovacuum turned off: > > ht

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-12 Thread Mark Wong
I thought I'd run a couple of tests to see if it would be helpful against CVS from Aug 3, 2005. Here's a run with autovacuum turned off: http://www.testing.osdl.org/projects/dbt2dev/results/dev4-015/42/ 5186.55 notpm Autvacuum on with default settings: http://www.testing.osdl.org/projects/dbt2dev

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-11 Thread Alvaro Herrera
On Thu, Aug 11, 2005 at 05:13:15PM -0400, Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Updated this patch again: > > Applied with minor tweaks --- mostly, fixing it so the custom cost > settings are applied for ANALYZE as well as VACUUM. Ok, cool, thanks. I think this complet

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-11 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Updated this patch again: > - vacuum_cost_delay and vacuum_cost_limit can be set per table, as well > as globally with autovacuum_vacuum_cost_{limit,delay} > > - pgstat is reset if recovery is required > > - pgstat reset at postmaster start is disable

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-10 Thread Alvaro Herrera
Updated this patch again: > > - vacuum_cost_delay and vacuum_cost_limit can be set per table, as well > > as globally with autovacuum_vacuum_cost_{limit,delay} > > > > - pgstat is reset if recovery is required > > > > - pgstat reset at postmaster start is disabled by default > > - Xid-wraparo

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-09 Thread Alvaro Herrera
On Tue, Aug 09, 2005 at 11:24:40PM -0400, Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Hmm. I have a patch for this, but now that it's ready, I wonder if it's > > really needed. If I understand vacuum_set_xid_limits() correctly, it's > > very difficult for the vacuumxid to be

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-09 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Hmm. I have a patch for this, but now that it's ready, I wonder if it's > really needed. If I understand vacuum_set_xid_limits() correctly, it's > very difficult for the vacuumxid to be far behind the freeze limit. Umm ... they can be close together,

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-09 Thread Alvaro Herrera
On Fri, Jul 29, 2005 at 03:33:09PM -0400, Tom Lane wrote: > It occurs to me that vacuuming to prevent XID wraparound is not the only > reason to do DB-wide vacuums: we also need to keep > pg_database.datvacuumxid from getting too old, else we will have > problems with clog bloat. We may need to r

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-08 Thread Alvaro Herrera
On Sun, Jul 31, 2005 at 07:36:36PM -0400, Alvaro Herrera wrote: Updated this patch: > - vacuum_cost_delay and vacuum_cost_limit can be set per table, as well > as globally with autovacuum_vacuum_cost_{limit,delay} > > - pgstat is reset if recovery is required > > - pgstat reset at postmaster

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-01 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: >>> - Xid-wraparound VACUUM is now FULL without ANALYZE >> >> Am I right in my assumption that this VACUUM FULL can happen for any >> database, not just a template database? > Ah, right. I think it would be OK if we made it FULL only for > datallowcon=

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-01 Thread Alvaro Herrera
On Mon, Aug 01, 2005 at 09:55:11AM +0200, Michael Paesold wrote: > Alvaro Herrera wrote: > > >Here is another patch for autovacuum: > ... > >- Xid-wraparound VACUUM is now FULL without ANALYZE > > Am I right in my assumption that this VACUUM FULL can happen for any > database, not just a templat

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-08-01 Thread Michael Paesold
Alvaro Herrera wrote: Here is another patch for autovacuum: ... - Xid-wraparound VACUUM is now FULL without ANALYZE Am I right in my assumption that this VACUUM FULL can happen for any database, not just a template database? I think this is a bad idea. Vacuum full is not an option for our

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-31 Thread Alvaro Herrera
Here is another patch for autovacuum: - vacuum_cost_delay and vacuum_cost_limit can be set per table, as well as globally with autovacuum_vacuum_cost_{limit,delay} - pgstat is reset if recovery is required - pgstat reset at postmaster start is disabled by default - Xid-wraparound VACUUM is no

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-31 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > On Fri, Jul 29, 2005 at 03:33:09PM -0400, Tom Lane wrote: >> It occurs to me that vacuuming to prevent XID wraparound is not the only >> reason to do DB-wide vacuums: we also need to keep >> pg_database.datvacuumxid from getting too old, else we will hav

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-31 Thread Alvaro Herrera
On Fri, Jul 29, 2005 at 03:33:09PM -0400, Tom Lane wrote: > It occurs to me that vacuuming to prevent XID wraparound is not the only > reason to do DB-wide vacuums: we also need to keep > pg_database.datvacuumxid from getting too old, else we will have > problems with clog bloat. We may need to r

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-30 Thread Alvaro Herrera
On Sat, Jul 30, 2005 at 10:57:15AM -0400, Bruce Momjian wrote: > Tom Lane wrote: > > Bruce Momjian writes: > > >>> I think you should at least add an autovacuum specific value for > > >>> "vacuum_cost_delay" because it turns cost-based vacuum delay on or off. > > > > > I am thinking we should mo

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-30 Thread Tom Lane
Bruce Momjian writes: >>> I think you should at least add an autovacuum specific value for >>> "vacuum_cost_delay" because it turns cost-based vacuum delay on or off. > I am thinking we should move ahead with what we have now, suggest the > work-arounds, and thensee what use-cases we have for it

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-30 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > >>> I think you should at least add an autovacuum specific value for > >>> "vacuum_cost_delay" because it turns cost-based vacuum delay on or off. > > > I am thinking we should move ahead with what we have now, suggest the > > work-arounds, and thensee w

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-30 Thread Bruce Momjian
Tom Lane wrote: > "Michael Paesold" <[EMAIL PROTECTED]> writes: > > Alvaro Herrera wrote: > >> I still haven't added custom cost-based delays, but I don't see that as > >> a showstopper for removing it. I just went through the CVS log and I > >> don't see anything else that applies. > > > I think

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-30 Thread Tom Lane
"Michael Paesold" <[EMAIL PROTECTED]> writes: > Alvaro Herrera wrote: >> I still haven't added custom cost-based delays, but I don't see that as >> a showstopper for removing it. I just went through the CVS log and I >> don't see anything else that applies. > I think you should at least add an au

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-30 Thread Michael Paesold
Alvaro Herrera wrote: I still haven't added custom cost-based delays, but I don't see that as a showstopper for removing it. I just went through the CVS log and I don't see anything else that applies. I think you should at least add an autovacuum specific value for "vacuum_cost_delay" becaus

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-29 Thread Alvaro Herrera
On Fri, Jul 29, 2005 at 05:46:11PM -0400, Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > >> Also the patch seemed to be missing diffs for header files? > > > Damn, I generated the diff from within src/backend instead of the root > > :-( Sorry for the inconvenience. > > No problem

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-29 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: >> Also the patch seemed to be missing diffs for header files? > Damn, I generated the diff from within src/backend instead of the root > :-( Sorry for the inconvenience. No problem --- reverse-engineering the changes to function declarations was simple

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-29 Thread Alvaro Herrera
On Fri, Jul 29, 2005 at 03:33:09PM -0400, Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Updated this patch: > > - The stat collector is modified so as to keep shared relations separate > > from regular ones. Autovacuum understands this. > > [etc] > > Applied with some fixes

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-29 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Updated this patch: > - The stat collector is modified so as to keep shared relations separate > from regular ones. Autovacuum understands this. > [etc] Applied with some fixes --- you had broken the reporting of statistics for shared tables, for o

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-29 Thread Alvaro Herrera
On Fri, Jul 29, 2005 at 11:19:34AM -0400, Tom Lane wrote: > BTW, is there still any reason not to remove the contrib/pg_autovacuum > directory from CVS? I still haven't added custom cost-based delays, but I don't see that as a showstopper for removing it. I just went through the CVS log and I don

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-29 Thread Tom Lane
BTW, is there still any reason not to remove the contrib/pg_autovacuum directory from CVS? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/f

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-27 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Ok, so there's a reason for having a manual stat-reset. However what's > the rationale for cleaning stats at postmaster start? In fact I think > it's actively bad because you lose any data you had before postmaster > stop/crash. We probably *should* d

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-27 Thread Alvaro Herrera
On Mon, Jul 25, 2005 at 09:31:15AM +0800, Christopher Kings-Lynne wrote: > >> We have to consider what > >> happens at stat reset -- AFAICS there's no problem, because as soon as > >> the table sees some activity, it will be picked up by pgstat. > >> However, it would be bad if stats are reset righ

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-26 Thread Matthew T. O'Connor
Alvaro Herrera wrote: Two comments still apply: - I haven't done anything yet w.r.t. the custom vacuum_delay nor sleep scale factor. I don't think we need the sleep scaling factor. Before we had vacuum delay settings, it might have been useful as a means of throttling down the impact of

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-25 Thread Alvaro Herrera
On Thu, Jul 14, 2005 at 10:52:56AM -0400, Tom Lane wrote: > I've applied Alvaro's latest integrated-autovacuum patch. There are > still a number of loose ends to be dealt with before beta, though: Updated this patch: - The stat collector is modified so as to keep shared relations separate from

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-25 Thread Matthew T. O'Connor
Tom Lane wrote: "Matthew T. O'Connor" writes: I don't know either, but this brings up another question. Stats wraparound. We'll all be safely dead, for one thing ;-) At one update per nanosecond, it'd take approximately 300 years to wrap a 64-bit counter. Somehow I don't have a probl

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-24 Thread Tom Lane
"Matthew T. O'Connor" writes: > I don't know either, but this brings up another question. Stats > wraparound. The n_tup_ins/upd/del columns in the stats system are > defined as bigint, what happens when the total number of upd for example > exceeds the capacity for bigint, or overflows to neg

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-24 Thread Matthew T. O'Connor
Alvaro Herrera wrote: On Sun, Jul 24, 2005 at 02:33:38PM -0400, Tom Lane wrote: Hmm, I wonder whether the minimum shouldn't be 10. Or even 60. It's ok with me. What do other people think? Effectiely, this is going to be the minimum amount of "down time" for autovacuum between

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-24 Thread Christopher Kings-Lynne
We have to consider what happens at stat reset -- AFAICS there's no problem, because as soon as the table sees some activity, it will be picked up by pgstat. However, it would be bad if stats are reset right after some heavy activity on a table. Maybe the only thing we need is documentation.

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-24 Thread Alvaro Herrera
On Sun, Jul 24, 2005 at 02:33:38PM -0400, Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > - pg_statistic is completely ignored. > > ... pg_statistic still needs vacuuming, surely. It's only ANALYZE > that you can/should skip for it. Sorry, yes, it's ignored only for analyze. >

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-24 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > - pg_statistic is completely ignored. ... pg_statistic still needs vacuuming, surely. It's only ANALYZE that you can/should skip for it. > - The postmaster's main loop sleeps Min(60, autovacuum_naptime), in > order to be able to pick naptimes smalle

Re: [PATCHES] [HACKERS] Autovacuum loose ends

2005-07-24 Thread Alvaro Herrera
On Thu, Jul 14, 2005 at 10:52:56AM -0400, Tom Lane wrote: > I've applied Alvaro's latest integrated-autovacuum patch. There are > still a number of loose ends to be dealt with before beta, though: Ok, here's a patch that deals with some of this: - The stat collector is modified so as to keep sha