Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Simon Riggs
On Sun, 2007-01-21 at 14:26 -0600, Jim C. Nasby wrote: On Sun, Jan 21, 2007 at 11:39:45AM +, Heikki Linnakangas wrote: Russell Smith wrote: Strange idea that I haven't researched, Given Vacuum can't be run in a transaction, it is possible at a certain point to quit the current

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Heikki Linnakangas
Russell Smith wrote: 2. Index cleanup is the most expensive part of vacuum. So doing a partial vacuum actually means more I/O as you have to do index cleanup more often. I don't think that's usually the case. Index(es) are typically only a fraction of the size of the table, and since 8.2 we

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Bruce Momjian
Heikki Linnakangas wrote: Russell Smith wrote: 2. Index cleanup is the most expensive part of vacuum. So doing a partial vacuum actually means more I/O as you have to do index cleanup more often. I don't think that's usually the case. Index(es) are typically only a fraction of the

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Heikki Linnakangas
Bruce Momjian wrote: Heikki Linnakangas wrote: Russell Smith wrote: 2. Index cleanup is the most expensive part of vacuum. So doing a partial vacuum actually means more I/O as you have to do index cleanup more often. I don't think that's usually the case. Index(es) are typically only a

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Bruce Momjian
Heikki Linnakangas wrote: Bruce Momjian wrote: Heikki Linnakangas wrote: Russell Smith wrote: 2. Index cleanup is the most expensive part of vacuum. So doing a partial vacuum actually means more I/O as you have to do index cleanup more often. I don't think that's usually the case.

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Simon Riggs
On Mon, 2007-01-22 at 12:18 -0500, Bruce Momjian wrote: Heikki Linnakangas wrote: In any case, for the statement Index cleanup is the most expensive part of vacuum to be true, you're indexes would have to take up 2x as much space as the heap, since the heap is scanned twice. I'm sure

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Gregory Stark
Bruce Momjian [EMAIL PROTECTED] writes: I agree it index cleanup isn't 50% of vacuum. I was trying to figure out how small, and it seems about 15% of the total table, which means if we have bitmap vacuum, we can conceivably reduce vacuum load by perhaps 80%, assuming 5% of the table is

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Bruce Momjian
Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to 10% dirty? I realize the vacuum is going to take a lot of time, but vacuuming to relaim 3% three times seems like it is

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Gregory Stark
Bruce Momjian [EMAIL PROTECTED] writes: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to 10% dirty? I realize the vacuum is going to take a lot of time, but

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Simon Riggs
On Mon, 2007-01-22 at 13:27 -0500, Bruce Momjian wrote: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to 10% dirty? I realize the vacuum is going to take a lot of

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Alvaro Herrera
Bruce Momjian wrote: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to 10% dirty? I realize the vacuum is going to take a lot of time, but vacuuming to relaim 3%

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Richard Huxton
Bruce Momjian wrote: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to 10% dirty? I realize the vacuum is going to take a lot of time, but vacuuming to relaim 3% three

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Heikki Linnakangas
Gregory Stark wrote: Bruce Momjian [EMAIL PROTECTED] writes: I agree it index cleanup isn't 50% of vacuum. I was trying to figure out how small, and it seems about 15% of the total table, which means if we have bitmap vacuum, we can conceivably reduce vacuum load by perhaps 80%, assuming 5%

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Heikki Linnakangas
Kenneth Marshall wrote: On Mon, Jan 22, 2007 at 06:42:09PM +, Simon Riggs wrote: Hold that thought! Read Heikki's Piggyback VACUUM idea on new thread... There may be other functions that could leverage a similar sort of infrastructure. For example, a long DB mining query could be

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Bruce Momjian
Alvaro Herrera wrote: Bruce Momjian wrote: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to 10% dirty? I realize the vacuum is going to take a lot of time, but

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Joris Dobbelsteen
Browne; pgsql-general@postgresql.org; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] [GENERAL] Autovacuum Improvements Bruce Momjian [EMAIL PROTECTED] writes: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Kenneth Marshall
On Mon, Jan 22, 2007 at 06:42:09PM +, Simon Riggs wrote: On Mon, 2007-01-22 at 13:27 -0500, Bruce Momjian wrote: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Kenneth Marshall
On Mon, Jan 22, 2007 at 07:24:20PM +, Heikki Linnakangas wrote: Kenneth Marshall wrote: On Mon, Jan 22, 2007 at 06:42:09PM +, Simon Riggs wrote: Hold that thought! Read Heikki's Piggyback VACUUM idea on new thread... There may be other functions that could leverage a similar sort of

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Steve Atkins
On Jan 22, 2007, at 11:16 AM, Richard Huxton wrote: Bruce Momjian wrote: Yep, agreed on the random I/O issue. The larger question is if you have a huge table, do you care to reclaim 3% of the table size, rather than just vacuum it when it gets to 10% dirty? I realize the vacuum is

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-22 Thread Martijn van Oosterhout
On Mon, Jan 22, 2007 at 05:51:53PM +, Gregory Stark wrote: Actually no. A while back I did experiments to see how fast reading a file sequentially was compared to reading the same file sequentially but skipping x% of the blocks randomly. The results were surprising (to me) and depressing.

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-21 Thread Heikki Linnakangas
Russell Smith wrote: Strange idea that I haven't researched, Given Vacuum can't be run in a transaction, it is possible at a certain point to quit the current transaction and start another one. There has been much chat and now a TODO item about allowing multiple vacuums to not starve small

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-21 Thread Simon Riggs
On Sat, 2007-01-20 at 09:41 +1100, Russell Smith wrote: Darcy Buskermolen wrote: [snip] Another thought, is it at all possible to do a partial vacuum? ie spend the next 30 minutes vacuuming foo table, and update the fsm with what hew have learned over the 30 mins, even if we

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-21 Thread Martijn van Oosterhout
On Sun, Jan 21, 2007 at 12:24:38PM +, Simon Riggs wrote: Partial vacuum would still be possible if you remembered where you got to in the VACUUM and then started from that same point next time. It could then go to the end of the table and wrap back around. ISTM the Dead Space Map would

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-21 Thread Jim C. Nasby
On Sun, Jan 21, 2007 at 11:39:45AM +, Heikki Linnakangas wrote: Russell Smith wrote: Strange idea that I haven't researched, Given Vacuum can't be run in a transaction, it is possible at a certain point to quit the current transaction and start another one. There has been much chat and

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-19 Thread Simon Riggs
On Tue, 2007-01-16 at 07:16 -0800, Darcy Buskermolen wrote: On Tuesday 16 January 2007 06:29, Alvaro Herrera wrote: elein wrote: Have you made any consideration of providing feedback on autovacuum to users? Right now we don't even know what tables were vacuumed when and what was

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-19 Thread Darcy Buskermolen
On Friday 19 January 2007 01:47, Simon Riggs wrote: On Tue, 2007-01-16 at 07:16 -0800, Darcy Buskermolen wrote: On Tuesday 16 January 2007 06:29, Alvaro Herrera wrote: elein wrote: Have you made any consideration of providing feedback on autovacuum to users? Right now we don't even

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-19 Thread Bruce Momjian
Added to TODO: o Allow multiple vacuums so large tables do not starve small tables http://archives.postgresql.org/pgsql-general/2007-01/msg00031.php o Improve control of auto-vacuum http://archives.postgresql.org/pgsql-hackers/2006-12/msg00876.php

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-19 Thread Russell Smith
Darcy Buskermolen wrote: [snip] Another thought, is it at all possible to do a partial vacuum? ie spend the next 30 minutes vacuuming foo table, and update the fsm with what hew have learned over the 30 mins, even if we have not done a full table scan ? There was a proposal for this, but

Re: [GENERAL] Autovacuum Improvements

2007-01-16 Thread Alvaro Herrera
Simon Riggs wrote: Perhaps we should focus on the issues that might result, so that we address those before we spend time on the details of the user interface. Can we deadlock or hang from running multiple autovacuums? If you were to run multiple autovacuum processes the way they are today,

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-16 Thread Alvaro Herrera
elein wrote: Have you made any consideration of providing feedback on autovacuum to users? Right now we don't even know what tables were vacuumed when and what was reaped. This might actually be another topic. I'd like to hear other people's opinions on Darcy Buskermolen proposal to have a

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-16 Thread Darcy Buskermolen
On Tuesday 16 January 2007 06:29, Alvaro Herrera wrote: elein wrote: Have you made any consideration of providing feedback on autovacuum to users? Right now we don't even know what tables were vacuumed when and what was reaped. This might actually be another topic. I'd like to hear other

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-16 Thread Matthew T. O'Connor
Alvaro Herrera wrote: I'd like to hear other people's opinions on Darcy Buskermolen proposal to have a log table, on which we'd register what did we run, at what time, how long did it last, how many tuples did it clean, etc. I feel having it on the regular text log is useful but it's not good

Re: [GENERAL] Autovacuum Improvements

2007-01-15 Thread Pavan Deolasee
Simon Riggs wrote: On Fri, 2006-12-29 at 20:25 -0300, Alvaro Herrera wrote: Christopher Browne wrote: Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table queue, and tables above

Re: [GENERAL] Autovacuum Improvements

2007-01-15 Thread Alban Hertroys
Pavan Deolasee wrote: Simon Riggs wrote: On Fri, 2006-12-29 at 20:25 -0300, Alvaro Herrera wrote: Christopher Browne wrote: Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table

Re: [GENERAL] Autovacuum Improvements

2007-01-14 Thread Pavan Deolasee
Simon Riggs wrote: On Fri, 2006-12-29 at 20:25 -0300, Alvaro Herrera wrote: Christopher Browne wrote: Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table queue, and tables above

Re: [HACKERS] [GENERAL] Autovacuum Improvements

2007-01-13 Thread elein
On Fri, Jan 12, 2007 at 07:33:05PM -0300, Alvaro Herrera wrote: Simon Riggs wrote: Some feedback from initial testing is that 2 queues probably isn't enough. If you have tables with 100s of blocks and tables with millions of blocks, the tables in the mid-range still lose out. So I'm

Re: [GENERAL] Autovacuum Improvements

2007-01-12 Thread Simon Riggs
On Fri, 2006-12-29 at 20:25 -0300, Alvaro Herrera wrote: Christopher Browne wrote: Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table queue, and tables above that size go to

Re: [GENERAL] Autovacuum Improvements

2007-01-12 Thread Alvaro Herrera
Simon Riggs wrote: Some feedback from initial testing is that 2 queues probably isn't enough. If you have tables with 100s of blocks and tables with millions of blocks, the tables in the mid-range still lose out. So I'm thinking that a design with 3 queues based upon size ranges, plus the

Re: [GENERAL] Autovacuum Improvements

2007-01-12 Thread Simon Riggs
On Fri, 2007-01-12 at 19:33 -0300, Alvaro Herrera wrote: Alvaro, have you completed your design? No, I haven't, and the part that's missing is precisely the queues stuff. I think I've been delaying posting it for too long, and that is harmful because it makes other people waste time

Re: [GENERAL] Autovacuum Improvements

2007-01-10 Thread Joris Dobbelsteen
-Original Message- From: Alvaro Herrera [mailto:[EMAIL PROTECTED] Sent: dinsdag 9 januari 2007 22:18 To: Joris Dobbelsteen Cc: Chris Browne; pgsql-general@postgresql.org Subject: Re: [GENERAL] Autovacuum Improvements Joris Dobbelsteen wrote: Now we have at least one different model

Re: [GENERAL] Autovacuum Improvements

2007-01-10 Thread Christopher Browne
A long time ago, in a galaxy far, far away, [EMAIL PROTECTED] (Csaba Nagy) wrote: On Mon, 2007-01-08 at 22:29, Chris Browne wrote: [snip] Based on the three policies I've seen, it could make sense to assign worker policies: 1. You have a worker that moves its way through the queue in some

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Csaba Nagy
On Mon, 2007-01-08 at 22:29, Chris Browne wrote: [snip] Based on the three policies I've seen, it could make sense to assign worker policies: 1. You have a worker that moves its way through the queue in some sort of sequential order, based on when the table is added to the queue, to

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Joris Dobbelsteen
-Original Message- From: Alvaro Herrera [mailto:[EMAIL PROTECTED] Sent: dinsdag 9 januari 2007 3:43 To: Joris Dobbelsteen Cc: Chris Browne; pgsql-general@postgresql.org Subject: Re: [GENERAL] Autovacuum Improvements Joris Dobbelsteen wrote: Why not collect some information from live

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Matthew T. O'Connor
Csaba Nagy wrote: Other thing, how will the vacuum queue be populated ? Or the queue here means nothing, all workers will always go through all tables to pick one based on their own criteria ? My concern here is that the current way of checking 1 DB per minute is not going to work with

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Csaba Nagy
On Tue, 2007-01-09 at 17:31, Matthew T. O'Connor wrote: Without getting into all the details, the autovacuum naptime is a GUC variable right now, so it can be much more frequent than the current default which is 60 seconds. Hmm, for some reason I thought the granularity is minutes, but it is

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Csaba Nagy
On Tue, 2007-01-09 at 17:36, Csaba Nagy wrote: On Tue, 2007-01-09 at 17:31, Matthew T. O'Connor wrote: Without getting into all the details, the autovacuum naptime is a GUC variable right now, so it can be much more frequent than the current default which is 60 seconds. Hmm, for some

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Matthew T. O'Connor
Csaba Nagy wrote: On Tue, 2007-01-09 at 17:36, Csaba Nagy wrote: On Tue, 2007-01-09 at 17:31, Matthew T. O'Connor wrote: Without getting into all the details, the autovacuum naptime is a GUC variable right now, so it can be much more frequent than the current default which is 60

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Alvaro Herrera
Joris Dobbelsteen wrote: Now we have at least one different model, lets mix in other captures and situations. So it cannot be done with only YOUR data, I fully agree. But if you have sufficient data you can find the generalization of the model to make it work (resonable) in sufficient

Re: [GENERAL] Autovacuum Improvements

2007-01-09 Thread Matthew T. O'Connor
Alvaro Herrera wrote: To be frank, I'm not sure I understand what you're saying here. I'm sure more analysis is good; that's easy to agree with. However, I don't want to be trapped in a design that's too hard to implement, or too hard for DBAs to manage. +1 There have been proposals to

Re: [GENERAL] Autovacuum Improvements

2007-01-08 Thread Csaba Nagy
On Sun, 2006-12-24 at 03:03, Christopher Browne wrote: [snip] Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table queue, and tables above that size go to the big table queue. That

Re: [GENERAL] Autovacuum Improvements

2007-01-08 Thread Chris Browne
[EMAIL PROTECTED] (Csaba Nagy) writes: On Sun, 2006-12-24 at 03:03, Christopher Browne wrote: [snip] Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table queue, and tables above that

Re: [GENERAL] Autovacuum Improvements

2007-01-08 Thread Joris Dobbelsteen
willing of capturing it? Who is willing to do some analysis... Just my EUR 0.02... - Joris -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Chris Browne Sent: maandag 8 januari 2007 22:30 To: pgsql-general@postgresql.org Subject: Re: [GENERAL] Autovacuum

Re: [GENERAL] Autovacuum Improvements

2007-01-08 Thread Alvaro Herrera
Joris Dobbelsteen wrote: Why not collect some information from live databases and perform some analysis on it? We already capture and utilize operational data from databases: for each table, how many tuples there are, how many tuples have been inserted, deleted, updated. We already analyse it

Re: [GENERAL] Autovacuum Improvements

2006-12-31 Thread Christopher Browne
A long time ago, in a galaxy far, far away, [EMAIL PROTECTED] (Alvaro Herrera) wrote: Christopher Browne wrote: Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table queue, and

Re: [GENERAL] Autovacuum Improvements

2006-12-29 Thread Alvaro Herrera
Christopher Browne wrote: Seems to me that you could get ~80% of the way by having the simplest 2 queue implementation, where tables with size some threshold get thrown at the little table queue, and tables above that size go to the big table queue. That should keep any small tables from

Re: [GENERAL] Autovacuum Improvements

2006-12-27 Thread Scott Ribe
Having a mechanism that requires enormous DBA effort and where there is considerable risk of simple configuration errors that will be hard to notice may not be the best kind of feature :-). Why not? It seems to have worked remarkably well for the market leader ;-) -- Scott Ribe [EMAIL

Re: [GENERAL] Autovacuum Improvements

2006-12-23 Thread Christopher Browne
After takin a swig o' Arrakan spice grog, [EMAIL PROTECTED] (Csaba Nagy) belched out: On Thu, 2006-12-21 at 18:41, Alvaro Herrera wrote: From all the discussion here I think the most benefit would result from a means to assign tables to different categories, and set up separate autovacuum

Re: [GENERAL] Autovacuum Improvements

2006-12-22 Thread Csaba Nagy
On Thu, 2006-12-21 at 18:41, Alvaro Herrera wrote: From all the discussion here I think the most benefit would result from a means to assign tables to different categories, and set up separate autovacuum rules per category (be it time window when vacuuming is allowed, autovacuum processes

Re: [GENERAL] Autovacuum Improvements

2006-12-21 Thread Matthew T. O'Connor
Russell Smith wrote: Alvaro Herrera wrote: I intend to work on the maintenance window idea for 8.3. I'm not sure if I'll be able to introduce the worker process stuff in there as well. I actually haven't done much design on the stuff so I can't say. What does a maintenance window mean? I

Re: [GENERAL] Autovacuum Improvements

2006-12-21 Thread Csaba Nagy
On Thu, 2006-12-21 at 18:03, Matthew T. O'Connor wrote: The maintenance window design as I understand it (Alvaro chime in if I get this wrong) is that we will be able to specify blocks of time that are assigned specific autovacuum settings. For example we might define a maintenance window

Re: [GENERAL] Autovacuum Improvements

2006-12-21 Thread Scott Ribe
I intend to work on the maintenance window idea for 8.3. I'm not sure if I'll be able to introduce the worker process stuff in there as well. I actually haven't done much design on the stuff so I can't say. Something to consider, per-day maintenance windows, where Sat Sun could have 24-hour

Re: [GENERAL] Autovacuum Improvements

2006-12-21 Thread Alvaro Herrera
Csaba Nagy wrote: On Thu, 2006-12-21 at 18:03, Matthew T. O'Connor wrote: The maintenance window design as I understand it (Alvaro chime in if I get this wrong) is that we will be able to specify blocks of time that are assigned specific autovacuum settings. For example we might define

Re: [GENERAL] Autovacuum Improvements

2006-12-20 Thread Joris Dobbelsteen
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Matthew O'Connor Sent: woensdag 20 december 2006 2:53 To: Glen Parker; Postgres General Subject: Re: [GENERAL] Autovacuum Improvements Alvaro Herrera wrote: [snip] I would go one step further and suggest

Re: [GENERAL] Autovacuum Improvements

2006-12-20 Thread Peter Childs
On 19/12/06, Chris Browne [EMAIL PROTECTED] wrote: matthew@zeut.net (Matthew O'Connor) writes: 2) Once we can have multiple autovacuum workers: Create the concept of hot tables that require more attention and should never be ignored for more that X minutes, perhaps have one autovacuum worker

Re: [GENERAL] Autovacuum Improvements

2006-12-20 Thread Alban Hertroys
Glen Parker wrote: Brandon Aiken wrote: You're saying that the dirtyness of a table is proportional to when you plan on vacuuming it next. The dirtiness of a table should most certainly have an effect on when it gets vacuumed in relation to other tables. If dirtiness could be rated, then

Re: [GENERAL] Autovacuum Improvements

2006-12-20 Thread Glen Parker
Joris Dobbelsteen wrote: Alvaro Herrera wrote: [snip] I would go one step further and suggest that when the maintenance window system gets completed we give it a default setup of midnight to 6AM or something like that. Do not do that... I agree. There is nothing very special about that

Re: [GENERAL] Autovacuum Improvements

2006-12-20 Thread Russell Smith
Alvaro Herrera wrote: Matthew O'Connor wrote: Glen Parker wrote: If it isn't there somewhere already, I would ask to add: 4) Expose all information used by autovacuum to form its decisions. You could argue that this is already there, although not easy to get at I suppose. But

Re: [GENERAL] Autovacuum Improvements

2006-12-20 Thread Glen Parker
Alvaro Herrera wrote: No, we currently don't expose the number of dead tuples which autovacuum uses. Patch submitted :-) ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail

[GENERAL] Autovacuum Improvements (was: Second attempt, roll your own autovacuum)

2006-12-19 Thread Matthew O'Connor
Alvaro Herrera wrote: Csaba Nagy wrote: Alternatively, perhaps a threshold so that a table is only considered for vacuum if: (table-size * overall-activity-in-last-hour) threshold Ideally you'd define your units appropriately so that you could just define threshold in postgresql.conf as

Re: [GENERAL] Autovacuum Improvements (was: Second attempt,

2006-12-19 Thread Bruce Momjian
Yes, I think there are these TODO items. I was waiting to see what additional replies there are before adding them. --- Matthew O'Connor wrote: Alvaro Herrera wrote: Csaba Nagy wrote: Alternatively, perhaps a

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
Bruce Momjian wrote: Yes, I think there are these TODO items. I was waiting to see what additional replies there are before adding them. Speaking of which, I was just looking at the TODO at: http://www.postgresql.org/docs/faqs.TODO.html and I think this item: * Improve xid wraparound

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Chris Browne
matthew@zeut.net (Matthew O'Connor) writes: 2) Once we can have multiple autovacuum workers: Create the concept of hot tables that require more attention and should never be ignored for more that X minutes, perhaps have one autovacuum worker per hot table? (What do people think of this?) One

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Chris Browne
matthew@zeut.net (Matthew O'Connor) writes: Bruce Momjian wrote: Yes, I think there are these TODO items. I was waiting to see what additional replies there are before adding them. Speaking of which, I was just looking at the TODO at: http://www.postgresql.org/docs/faqs.TODO.html and I

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Alvaro Herrera
Chris Browne wrote: matthew@zeut.net (Matthew O'Connor) writes: Bruce Momjian wrote: Yes, I think there are these TODO items. I was waiting to see what additional replies there are before adding them. Speaking of which, I was just looking at the TODO at:

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
1) Allow multiple autovacuum workers: Currently Autovacuum is only capable of ordering one vacuum command at a time, for most work loads this is sufficient but falls down when a hot (very actively updated table) goes unvacuumed for a long period of time because a large table is currently

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
Glen Parker wrote: If it isn't there somewhere already, I would ask to add: 4) Expose all information used by autovacuum to form its decisions. You could argue that this is already there, although not easy to get at I suppose. But all table threshold settings are available either in the

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Alvaro Herrera
Matthew O'Connor wrote: Glen Parker wrote: If it isn't there somewhere already, I would ask to add: 4) Expose all information used by autovacuum to form its decisions. You could argue that this is already there, although not easy to get at I suppose. But all table threshold settings are

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Erik Jones
Matthew O'Connor wrote: Glen Parker wrote: If it isn't there somewhere already, I would ask to add: Expose a very easy way to discover autovacuum's opinion about a particular table, for example table_needs_vacuum(oid), ignoring any time constraints that may be in place. This might be a

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Bruce Momjian
Alvaro Herrera wrote: Chris Browne wrote: matthew@zeut.net (Matthew O'Connor) writes: Bruce Momjian wrote: Yes, I think there are these TODO items. I was waiting to see what additional replies there are before adding them. Speaking of which, I was just looking at the TODO at:

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
Erik Jones wrote: Matthew O'Connor wrote: Glen Parker wrote: If it isn't there somewhere already, I would ask to add: Expose a very easy way to discover autovacuum's opinion about a particular table, for example table_needs_vacuum(oid), ignoring any time constraints that may be in place.

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
Alvaro Herrera wrote: 4) Expose all information used by autovacuum to form its decisions. You could argue that this is already there, although not easy to get at I suppose. But all table threshold settings are available either in the pg_autovacuum relation or the defaults via GUC variables,

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Alvaro Herrera
Glen Parker wrote: Alvaro Herrera wrote: 4) Expose all information used by autovacuum to form its decisions. You could argue that this is already there, although not easy to get at I suppose. But all table threshold settings are available either in the pg_autovacuum relation or the

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
Alvaro Herrera wrote: Matthew O'Connor wrote: Glen Parker wrote: 5) Expose a very easy way to discover autovacuum's opinion about a particular table, for example table_needs_vacuum(oid), ignoring any time constraints that may be in place. This might be a nice feature however in the presence

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
Glen Parker wrote: Erik Jones wrote: Matthew O'Connor wrote: Glen Parker wrote: If it isn't there somewhere already, I would ask to add: Expose a very easy way to discover autovacuum's opinion about a particular table, for example table_needs_vacuum(oid), ignoring any time constraints that

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Brandon Aiken
Parker Cc: Postgres general mailing list Subject: Re: [GENERAL] Autovacuum Improvements Glen Parker wrote: Erik Jones wrote: Matthew O'Connor wrote: Glen Parker wrote: If it isn't there somewhere already, I would ask to add: Expose a very easy way to discover autovacuum's opinion about

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
] [mailto:[EMAIL PROTECTED] On Behalf Of Matthew O'Connor Sent: Tuesday, December 19, 2006 5:37 PM To: Glen Parker Cc: Postgres general mailing list Subject: Re: [GENERAL] Autovacuum Improvements Glen Parker wrote: Erik Jones wrote: Matthew O'Connor wrote: Glen Parker wrote: If it isn't there somewhere

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
Matthew O'Connor wrote: No, how dirty a table isn't subjective, what is subjective is the question Does it need to be vacuumed?. A that is 1% dirty (to use your term) probably doesn't *need* to be vacuumed, but you might choose to vacuum it anyway at least you might at night when the system

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
Glen Parker wrote: Matthew O'Connor wrote: No, how dirty a table isn't subjective, what is subjective is the question Does it need to be vacuumed?. A that is 1% dirty (to use your term) probably doesn't *need* to be vacuumed, but you might choose to vacuum it anyway at least you might at

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Bruce Momjian
Matthew O'Connor wrote: Glen Parker wrote: Matthew O'Connor wrote: No, how dirty a table isn't subjective, what is subjective is the question Does it need to be vacuumed?. A that is 1% dirty (to use your term) probably doesn't *need* to be vacuumed, but you might choose to vacuum it

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
I wonder if the simple solution is to just have a cron script modify postgresql.conf and pg_ctl reload. That seems very flexible, or have two postgresql.conf files and move them into place via cron. I'd still prefer to vacuum on demand actually. Rather than hope that autovacuum hit all the

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
The idea that has been discussed in the past is the concept of maintenance windows, that is for any given period of time, you can set different vacuum thresholds. So at night you might make the thresholds very low so that nearly everything gets vacuumed but during the day you might only

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Alvaro Herrera
Glen Parker wrote: The idea that has been discussed in the past is the concept of maintenance windows, that is for any given period of time, you can set different vacuum thresholds. So at night you might make the thresholds very low so that nearly everything gets vacuumed but during the

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
Glen Parker wrote: I wonder if the simple solution is to just have a cron script modify postgresql.conf and pg_ctl reload. That seems very flexible, or have two postgresql.conf files and move them into place via cron. I'd still prefer to vacuum on demand actually. Rather than hope that

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Matthew O'Connor
Alvaro Herrera wrote: Glen Parker wrote: That really seems like something that, if it's powerful, would also be very complicated. If the autovacuum system could just call a user defined function, all the complexity could be dropped back into the admin's lap (which is fine with me :-). I

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Alvaro Herrera
Glen Parker wrote: I'd also like to use some of this information to issue reindex and cluster commands only when they're needed. In fact, on days when I cluster, there's no need whatsoever to also vacuum those tables. This is something that autovacuum won't do at all. Well, I'd rather

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
Alvaro Herrera wrote: Glen Parker wrote: I'd also like to use some of this information to issue reindex and cluster commands only when they're needed. In fact, on days when I cluster, there's no need whatsoever to also vacuum those tables. This is something that autovacuum won't do at all.

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Glen Parker
Brandon Aiken wrote: You're saying that the dirtyness of a table is proportional to when you plan on vacuuming it next. The dirtiness of a table should most certainly have an effect on when it gets vacuumed in relation to other tables. If dirtiness could be rated, then the list of

Re: [GENERAL] Autovacuum Improvements

2006-12-19 Thread Alvaro Herrera
Glen Parker wrote: Alvaro Herrera wrote: Glen Parker wrote: I'd also like to use some of this information to issue reindex and cluster commands only when they're needed. In fact, on days when I cluster, there's no need whatsoever to also vacuum those tables. This is something that