Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Joshua D. Drake
Josh Berkus wrote: Josh, Just my own two cents. First I am not knocking the work that has been on autovacuum. I am sure that it was a leap on its own to get it to work. However I will say that I just don't see the reason for it. I've personally seen at least a dozen user requests for

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Joshua D. Drake
Matthew T. O'Connor wrote: Joshua D. Drake wrote: Just my own two cents. First I am not knocking the work that has been on autovacuum. I am sure that it was a leap on its own to get it to work. However I will say that I just don't see the reason for it. The major reasons for autovacuum as

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Hans-Jürgen Schönig
* Reduces the total amount of time the system spends vacuuming since it only vacuums when needed. Can be easily done with cron. * Keeps stats up-to-date automatically Which can be done with cron * Eliminates newbie confusion RTFM * Eliminates one of the criticisms that the public

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Andreas Pflug
Qingqing Zhou wrote: Tom Lane [EMAIL PROTECTED] writes: Yeah --- a libpq-based solution is not what I think of as integrated at all, because it cannot do anything that couldn't be done by the existing external autovacuum process. About all you can buy there is having the postmaster spawn the

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Russell Smith
The major reasons for autovacuum as I see it are as follows: * Reduces administrative overhead having to keep track of what tables need to be vacuumed how often. Creates more overhead and thus reduces performance. Or reduces vacuum overhead because the vacuum strategy is much better

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Russell Smith
Added to TODO: * Create a bitmap of pages that need vacuuming Instead of sequentially scanning the entire table, have the background writer or some other process record pages that have expired rows, then VACUUM can look at just those pages rather than the entire table. In

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Russell Smith
4) Related to this, I guess, is that a user's FSM settings might be completely inappropriate. The 'Just read the manual' or 'Just read the logs' argument doesn't cut it, because the main argument for autovacuum in the backend is that people do not and will not. Agreed, it doesn't

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Russell Smith
On Fri, 17 Jun 2005 06:26 pm, Andreas Pflug wrote: Qingqing Zhou wrote: Tom Lane [EMAIL PROTECTED] writes: Yeah --- a libpq-based solution is not what I think of as integrated at all, because it cannot do anything that couldn't be done by the existing external autovacuum process. About

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Gavin Sherry
On Fri, 17 Jun 2005, Russell Smith wrote: 4) Related to this, I guess, is that a user's FSM settings might be completely inappropriate. The 'Just read the manual' or 'Just read the logs' argument doesn't cut it, because the main argument for autovacuum in the backend is that people do not

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Gavin Sherry
On Fri, 17 Jun 2005, Russell Smith wrote: Added to TODO: * Create a bitmap of pages that need vacuuming Instead of sequentially scanning the entire table, have the background writer or some other process record pages that have expired rows, then VACUUM can look at just

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Matthew T. O'Connor
Joshua D. Drake wrote: Josh Berkus wrote: I've personally seen at least a dozen user requests for autovacuum in the backend, and had this conversation about 1,100 times: NB: After a week, my database got really slow. Me: How often are you running VACUUM ANALYZE? NB: Running what? Can't

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Matthew T. O'Connor
Joshua D. Drake wrote: Matthew T. O'Connor wrote: The major reasons for autovacuum as I see it are as follows: * Reduces administrative overhead having to keep track of what tables need to be vacuumed how often. Creates more overhead and thus reduces performance. In the general case,

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Matthew T. O'Connor
Russell Smith wrote: * Reduces the total amount of time the system spends vacuuming since it only vacuums when needed. Can be easily done with cron. Can you do partial table vacuums with CRON? You can work out the smartest time to vacuum with cron? I thought it just scheduled

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Matthew T. O'Connor
Russell Smith wrote: On Fri, 17 Jun 2005 06:26 pm, Andreas Pflug wrote: Qingqing Zhou wrote: One reason of not using lib-pq is that this one has to wait for the completion of each vacuum (we don't has async execution in libpq right?), There *is* async execution in libpq, and

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Christopher Browne
[EMAIL PROTECTED] (Gavin Sherry) wrote: I guess the main point is, if something major like this ships in the backend it says to users that the problem has gone away. pg_autovacuum is a good contrib style solution: it addresses a problem users have and attempts to solve it the way other users

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Tom Lane
Matthew T. O'Connor matthew@zeut.net writes: ... People just didn't like including libpq into the backend for reasons I don't remember. One reason I can think of is that there would be global-symbol conflicts --- libpq has copies of some backend routines, but they are not identical. In any

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Rod Taylor
On Fri, 2005-06-17 at 00:03 -0700, Joshua D. Drake wrote: Matthew T. O'Connor wrote: Joshua D. Drake wrote: Just my own two cents. First I am not knocking the work that has been on autovacuum. I am sure that it was a leap on its own to get it to work. However I will say that I just

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Josh Berkus
Josh, Just so everyone knows from the get go here. I am purposely playing a little devils advocate. Well, please stop it. We discussed AV over a year ago when we ran out of time to integrate it with 8.0. This disucussion now is hindering any discussion of what needs to be *done* to

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Matthew T. O'Connor
Christopher Browne wrote: [EMAIL PROTECTED] (Gavin Sherry) wrote: I guess the main point is, if something major like this ships in the backend it says to users that the problem has gone away. pg_autovacuum is a good contrib style solution: it addresses a problem users have and attempts to

Re: [HACKERS] Autovacuum in the backend

2005-06-17 Thread Jim C. Nasby
On Fri, Jun 17, 2005 at 12:21:44PM -0400, Matthew T. O'Connor wrote: snip more stuff about how we need to track pages with dead tuples This will make VACUUM less painful, but it doesn't eliminate the need / desire for autovacuum. I agree this would be good, but I see it as a separate

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Gavin Sherry
On Thu, 16 Jun 2005, Matthew T. O'Connor wrote: Gavin Sherry wrote: I guess my main concern is that we'll have a solution to the problem of dead tuples which is only half way there. It is only an incremental improvement upon the contrib module and solves only one real problem: users do not

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Hans-Jürgen Schönig
2) By no fault of its own, autovacuum's level of granularity is the table level. For people dealing with non-trivial amounts of data (and we're not talking gigabytes or terabytes here), this is a serious drawback. Vacuum at peak times can cause very intense IO bursts -- even with the enhancements

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Magnus Hagander
If we do integrated AV, it should only be turned on by default at a relatively low level. And wasn't there an issue on Windows with AV not working? AFAIK, it works. But the fact that you need to provide it with a userid/password combo makes it a lot harder to install as a service. And it's

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Gavin Sherry
On Thu, 16 Jun 2005, [ISO-8859-1] Hans-Jürgen Schönig wrote: 2) By no fault of its own, autovacuum's level of granularity is the table level. For people dealing with non-trivial amounts of data (and we're not talking gigabytes or terabytes here), this is a serious drawback. Vacuum at peak

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Hans-Jrgen Schnig
Gavin Sherry wrote: On Thu, 16 Jun 2005, [ISO-8859-1] Hans-Jrgen Schnig wrote: 2) By no fault of its own, autovacuum's level of granularity is the table level. For people dealing with non-trivial amounts of data (and we're not talking gigabytes or terabytes here), this is a serious drawback.

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Dave Page
-Original Message- From: Magnus Hagander [mailto:[EMAIL PROTECTED] Sent: 16 June 2005 10:15 To: Josh Berkus; Gavin Sherry Cc: Bruce Momjian; pgsql-hackers@postgresql.org; Dave Page Subject: RE: [HACKERS] Autovacuum in the backend If we do integrated AV, it should only

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Andrew Dunstan
Gavin Sherry said: On Thu, 16 Jun 2005, [ISO-8859-1] Hans-Jrgen Schnig wrote: 2) By no fault of its own, autovacuum's level of granularity is the table level. For people dealing with non-trivial amounts of data (and we're not talking gigabytes or terabytes here), this is a serious

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Matthew T. O'Connor
Andrew Dunstan wrote: Gavin Sherry said: I think this gets away from my point a little. People with 2 TB tables can take care of themselves, as can people who've taken the time to partition their tables to speed up vacuum. I'm more concerned about the majority of people who fall in the

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Matthew T. O'Connor
Gavin Sherry wrote: On Thu, 16 Jun 2005, Matthew T. O'Connor wrote: Would you mind expounding on why you think autovacuum isn't suitable for the general public? I know it's not a silver bullet, but I think in general, it will be helpful for most people. As I said, this is largely

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Matthew T. O'Connor
Hans-Jrgen Schnig wrote: I completly agree with Gavin - integrating this kind of thing into the backend writer or integrate it with FSM would be the ideal solution. Yes AV should look at FSM data, and it will eventually. I'm not sure how you would integrate AV with the backend writer, but

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Steve Atkins
On Wed, Jun 15, 2005 at 09:07:30PM -0700, Josh Berkus wrote: Josh, Just my own two cents. First I am not knocking the work that has been on autovacuum. I am sure that it was a leap on its own to get it to work. However I will say that I just don't see the reason for it. I've personally

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Andreas Pflug
Matthew T. O'Connor wrote: Right, I think if VACUUM is improved than the semantics of AV in the backend might change, but I think there will always be a need for some maintenance, and a daemon that monitors the maintenance needs of your database and fires off appropriate maintenance

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Alvaro Herrera
On Thu, Jun 16, 2005 at 01:32:16AM -0400, Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: A question for interested parties. I'm thinking in handling the user/password issue by reading the flat files (the copies of pg_shadow, pg_database, etc). Er, what user/password issue?

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Alvaro Herrera
On Thu, Jun 16, 2005 at 04:20:34PM +1000, Gavin Sherry wrote: 2) By no fault of its own, autovacuum's level of granularity is the table level. For people dealing with non-trivial amounts of data (and we're not talking gigabytes or terabytes here), this is a serious drawback. Vacuum at peak

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Hans-Jürgen Schönig
Alvaro Herrera wrote: On Thu, Jun 16, 2005 at 04:20:34PM +1000, Gavin Sherry wrote: 2) By no fault of its own, autovacuum's level of granularity is the table level. For people dealing with non-trivial amounts of data (and we're not talking gigabytes or terabytes here), this is a serious

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Now, I'm hearing people don't like using libpq. Yeah --- a libpq-based solution is not what I think of as integrated at all, because it cannot do anything that couldn't be done by the existing external autovacuum process. About all you can buy there is

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Dave Page
: [HACKERS] Autovacuum in the backend Dave, i wonder if we should aim to have pgAgent in the backend which was one of the reasons why I considered to have it converted from C++ to pure C. In previous discussions on -hackers when ppl raised the idea of something like pgAgent being built

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Josh Berkus
Alvaro, coffee-with-cream vacuums. I tried this and now my Hoover makes this horrible noise and smokes. ;-) All: Seriously, all: when I said that users were asking for Autovac in the backend (AVitB), I wasn't talking just the newbies on #postgresql. I'm also talking companies like

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Josh Berkus
People, AVitB has been on the TODO list for 2 versions. There's been 2 years to question its position there. Now people are bringing up objections when there's no time for discussion left? This stinks. Hmmm ... to be specific, I'm referring to the objections to the *idea* of AVitB, not the

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Josh Berkus
Dave, In previous discussions on -hackers when ppl raised the idea of something like pgAgent being built into the backend, istm that the majority of people were against the idea. Well, you're up against the minimalist approach to core PostgreSQL there. It would pretty much *have* to be an

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Douglas McNaught
Josh Berkus josh@agliodbs.com writes: Seriously, all: when I said that users were asking for Autovac in the backend (AVitB), I wasn't talking just the newbies on #postgresql. I'm also talking companies like Hyperic, and whole groups like the postgresql.org.br. This is a feature that

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Matthew T. O'Connor
Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Now, I'm hearing people don't like using libpq. Yeah --- a libpq-based solution is not what I think of as integrated at all, because it cannot do anything that couldn't be done by the existing external autovacuum process.

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Tom Lane
Matthew T. O'Connor matthew@zeut.net writes: I don't think you can use a dump to determine who should be connected to next since you don't really know what happened since the last time you exited. What was a priority 5 or 10 minutes ago might not be a priority now. Well, the information

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Matthew T. O'Connor
Tom Lane wrote: Matthew T. O'Connor matthew@zeut.net writes: I don't think you can use a dump to determine who should be connected to next since you don't really know what happened since the last time you exited. What was a priority 5 or 10 minutes ago might not be a priority now.

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Hannu Krosing
On N, 2005-06-16 at 11:42 -0400, Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: ... Some people say keep it simple and have one process per cluster. I think they don't realize it's actually more complex, not the other way around. Agreed. If you aren't connected to a specific

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Andreas Pflug
Josh Berkus wrote: Dave, In previous discussions on -hackers when ppl raised the idea of something like pgAgent being built into the backend, istm that the majority of people were against the idea. Well, you're up against the minimalist approach to core PostgreSQL there. It would pretty

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Andrew Dunstan
Andreas Pflug wrote: We're having a growing zoo of daemons that can be regarded as tightly integrated server add-on processes (slony, autovac, pgAgent), and it would be really nice (say: win32 users are used to it, thus requiring it) to have a single point of control. Maybe a super

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Gavin Sherry
On Thu, 16 Jun 2005, Alvaro Herrera wrote: On Thu, Jun 16, 2005 at 04:20:34PM +1000, Gavin Sherry wrote: 2) By no fault of its own, autovacuum's level of granularity is the table level. For people dealing with non-trivial amounts of data (and we're not talking gigabytes or terabytes

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Andreas Pflug
Andrew Dunstan wrote: Andreas Pflug wrote: We're having a growing zoo of daemons that can be regarded as tightly integrated server add-on processes (slony, autovac, pgAgent), and it would be really nice (say: win32 users are used to it, thus requiring it) to have a single point of

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Tim Allen
Josh Berkus wrote: Alvaro, coffee-with-cream vacuums. I tried this and now my Hoover makes this horrible noise and smokes. ;-) Probably related to the quality of American coffee ;). All: Seriously, all: when I said that users were asking for Autovac in the backend (AVitB), I wasn't

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Qingqing Zhou
Tom Lane [EMAIL PROTECTED] writes: Yeah --- a libpq-based solution is not what I think of as integrated at all, because it cannot do anything that couldn't be done by the existing external autovacuum process. About all you can buy there is having the postmaster spawn the autovacuum process,

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Bruce Momjian
Gavin Sherry wrote: In January I was in Toronto with Jan, Tom and others and some ideas about vacuum were being discussed. The basic idea is that when we dirty pages we need we set a bit in a bitmap to say that the page has been dirty. A convenient place to do this is when we are writing dirty

Re: [HACKERS] Autovacuum in the backend

2005-06-16 Thread Thomas F. O'Connell
Gavin, For the record, I don't consider myself a PostgreSQL newbie, nor do I manage any 2 TB databases (much less tables), but I do have an unusual production use case: thousands ( 10,000) of tables, many of them inherited, and many of them with hundreds of thousands (a few with

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Hannu Krosing
On T, 2005-06-14 at 21:23 -0400, Bruce Momjian wrote: One goal for 8.1 is to move /contrib/pg_autovacuum in to the backend. I think it has to be done in four stages: o move it into the backend and have it start/stop automatically o move the autovacuum configuration parameters

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Hannu Krosing
On K, 2005-06-15 at 13:41 +0800, Qingqing Zhou wrote: Neil Conway [EMAIL PROTECTED] writes Wasn't the plan to rewrite pg_autovacuum to use the FSM rather than the stats collector? I don't understand. Currently the basic logic of pg_autovacuum is to use the pg_stat_all_tables numbers

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Qingqing Zhou
Qingqing Zhou [EMAIL PROTECTED] writes The start/stop routine is quite like Bgwriter. I just realized that the non-standard backend can't execute any SQL command. If so, how would the background pg_autovacuum do vacuum? It could be forked more like a standard backend, but this is obviously

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Bruce Momjian
Hannu Krosing wrote: On K, 2005-06-15 at 13:41 +0800, Qingqing Zhou wrote: Neil Conway [EMAIL PROTECTED] writes Wasn't the plan to rewrite pg_autovacuum to use the FSM rather than the stats collector? I don't understand. Currently the basic logic of pg_autovacuum is to use the

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Alvaro Herrera
On Wed, Jun 15, 2005 at 11:23:20AM +0800, Qingqing Zhou wrote: Who would like to get started on this? It seems pretty straight-forward. I'd like to give it a try. I'm on it. I have Matthew's patch, updated to current sources, and I'm working on cleaning it up to address all known

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Bruce Momjian
I am going to start working on it. I am concerned it is a big job. I will post questions as I find them, and the one below is a good one. --- Qingqing Zhou wrote: Qingqing Zhou [EMAIL PROTECTED] writes The

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Bruce Momjian
Alvaro Herrera wrote: On Wed, Jun 15, 2005 at 11:23:20AM +0800, Qingqing Zhou wrote: Who would like to get started on this? It seems pretty straight-forward. I'd like to give it a try. I'm on it. I have Matthew's patch, updated to current sources, and I'm working on cleaning it up

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Josh Berkus
Qingqing, add a parameter to let user pass in the configuration parameters: * autovacuum_command = -s 100 -S 1 ... um, can we have these as separate GUCs and not lumped together as a string? i.e.: autovacuum_frequency = 60 #seconds, 0 = disable autovacuum_vacuum_threshold = 200

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Bruce Momjian wrote: Hannu Krosing wrote: On K, 2005-06-15 at 13:41 +0800, Qingqing Zhou wrote: Neil Conway [EMAIL PROTECTED] writes Wasn't the plan to rewrite pg_autovacuum to use the FSM rather than the stats collector? Yes, those are step five. The TODO list has:

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Bruce Momjian wrote: I am going to start working on it. I am concerned it is a big job. I will post questions as I find them, and the one below is a good one. It is a big question, at least it's the main stumbling block I had, and it's the one that kept my work from being integrated

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Josh Berkus wrote: Qingqing, add a parameter to let user pass in the configuration parameters: * autovacuum_command = -s 100 -S 1 ... um, can we have these as separate GUCs and not lumped together as a string? i.e.: autovacuum_frequency = 60 #seconds, 0 = disable

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Hannu Krosing wrote: Can autovacuum yet be configured _not_ to run vacuum during some hours or above some load ? That is certainly a goal, hopefully it will get done for 8.1.The actual design I had in mind (based on prior discussion on hackers) is to allow a maintenance window that

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Bruce Momjian
Matthew T. O'Connor wrote: I don't think the current implementation if VACUUM can support that. I believe that all the work will get rolled back if gets canceled. Perhaps a decent solution would be to have autovacuum increase the vacuum delay settings dynamically based on system load

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Gavin Sherry
On Wed, 15 Jun 2005, Bruce Momjian wrote: I am going to start working on it. I am concerned it is a big job. I will post questions as I find them, and the one below is a good one. I'm wondering if effort is being misdirected here. I remember when Mark Wong at OSDL was running pg_autovacuum

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Christopher Kings-Lynne
um, can we have these as separate GUCs and not lumped together as a string? i.e.: autovacuum_frequency = 60 #seconds, 0 = disable autovacuum_vacuum_threshold = 200 autovacuum_vacuum_multiple = 0.5 autovacuum_analyze_threshold = 100 autovacuum_analyze_multiple = 0.4 AV should be disabled by

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Bruce Momjian
I totally agree. I think we know pg_autovacuum is just a short-term solution. What we need is for someone to take the lead in this. The TODO list has the ideas documented. There is no reason the background writer could not load the FSM directly with free pages.

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Alvaro Herrera
A question for interested parties. I'm thinking in handling the user/password issue by reading the flat files (the copies of pg_shadow, pg_database, etc). The only thing that I'd need to modify is add the datdba field to pg_database, so we can figure out an appropiate user for vacuuming each

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Alvaro Herrera
On Thu, Jun 16, 2005 at 11:07:31AM +1000, Gavin Sherry wrote: On Wed, 15 Jun 2005, Bruce Momjian wrote: I am going to start working on it. I am concerned it is a big job. I will post questions as I find them, and the one below is a good one. I'm wondering if effort is being

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Russell Smith
On Thu, 16 Jun 2005 12:54 pm, Alvaro Herrera wrote: On Thu, Jun 16, 2005 at 11:07:31AM +1000, Gavin Sherry wrote: On Wed, 15 Jun 2005, Bruce Momjian wrote: I am going to start working on it. I am concerned it is a big job. I will post questions as I find them, and the one below is

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Josh Berkus
Gavin, People, I'm wondering if effort is being misdirected here. I remember when Mark Wong at OSDL was running pg_autovacuum during a dbt run, he was seeing significant performance loss -- I think on the order of 30% to 40% (I will try and dig up a link to the results). It wasn't quite that

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Alvaro Herrera wrote: A question for interested parties. I'm thinking in handling the user/password issue by reading the flat files (the copies of pg_shadow, pg_database, etc). The only thing that I'd need to modify is add the datdba field to pg_database, so we can figure out an appropiate

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Gavin Sherry
On Wed, 15 Jun 2005, Alvaro Herrera wrote: On Thu, Jun 16, 2005 at 11:07:31AM +1000, Gavin Sherry wrote: On Wed, 15 Jun 2005, Bruce Momjian wrote: I am going to start working on it. I am concerned it is a big job. I will post questions as I find them, and the one below is a good

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Joshua D. Drake
The big leap will be to get it in the backend, which will as Chris KL suggested stop people shooting themselves in the foot. I'm not sure what strict rules are imposed by Feature freeze, but there may be time for others to make some improvements before 8.1. We have also looked at this for at

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Alvaro Herrera
On Wed, Jun 15, 2005 at 11:42:17PM -0400, Matthew T. O'Connor wrote: Alvaro Herrera wrote: A question for interested parties. I'm thinking in handling the user/password issue by reading the flat files (the copies of pg_shadow, pg_database, etc). The only thing that I'd need to modify is

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Josh Berkus
Alvaro, One issue I do have to deal with right now is how many autovacuum processes do we want to be running.  The current approach is to have one autovacuum process.  Two possible options would be to have one per database, and one per tablespace.  What do people think? I'd vote for one,

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Neil Conway
Alvaro Herrera wrote: One issue I do have to deal with right now is how many autovacuum processes do we want to be running. The current approach is to have one autovacuum process. Two possible options would be to have one per database, and one per tablespace. What do people think? Why do we

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Josh Berkus
Josh, Just my own two cents. First I am not knocking the work that has been on autovacuum. I am sure that it was a leap on its own to get it to work. However I will say that I just don't see the reason for it. I've personally seen at least a dozen user requests for autovacuum in the backend,

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Neil Conway
Josh Berkus wrote: Not that I don't agree that we need a less I/O intense alternative to VACUUM, but it seems unlikely that we could actually do this, or even agree on a spec, before feature freeze. I don't see the need to rush anything in before the feature freeze. Wheras integrated AV is

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Christopher Kings-Lynne
I've personally seen at least a dozen user requests for autovacuum in the backend, and had this conversation about 1,100 times: NB: After a week, my database got really slow. Me: How often are you running VACUUM ANALYZE? NB: Running what? Me too. Just hang out in #postgresql for a while :)

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Alvaro Herrera
On Thu, Jun 16, 2005 at 02:09:47PM +1000, Neil Conway wrote: Alvaro Herrera wrote: One issue I do have to deal with right now is how many autovacuum processes do we want to be running. The current approach is to have one autovacuum process. Two possible options would be to have one per

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Gavin Sherry wrote: I guess my main concern is that we'll have a solution to the problem of dead tuples which is only half way there. It is only an incremental improvement upon the contrib module and solves only one real problem: users do not read up on VACUUM or autovacuum. This is at the

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Joshua D. Drake wrote: Just my own two cents. First I am not knocking the work that has been on autovacuum. I am sure that it was a leap on its own to get it to work. However I will say that I just don't see the reason for it. The major reasons for autovacuum as I see it are as follows: *

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Neil Conway wrote: Josh Berkus wrote: Wheras integrated AV is something we *could* do, and is widely desired. I don't see why. IMHO the current autovacuum approach is far from optimal. If integrated autovacuum just means taking the same approach and building it into the backend, how does

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Alvaro Herrera wrote: On Thu, Jun 16, 2005 at 02:09:47PM +1000, Neil Conway wrote: Alvaro Herrera wrote: One issue I do have to deal with right now is how many autovacuum processes do we want to be running. The current approach is to have one autovacuum process. Two possible options

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Matthew T. O'Connor
Neil Conway wrote: Alvaro Herrera wrote: One issue I do have to deal with right now is how many autovacuum processes do we want to be running. The current approach is to have one autovacuum process. Two possible options would be to have one per database, and one per tablespace. What do

Re: [HACKERS] Autovacuum in the backend

2005-06-15 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: A question for interested parties. I'm thinking in handling the user/password issue by reading the flat files (the copies of pg_shadow, pg_database, etc). Er, what user/password issue? Context please. The only thing that I'd need to modify is add

[HACKERS] Autovacuum in the backend

2005-06-14 Thread Bruce Momjian
One goal for 8.1 is to move /contrib/pg_autovacuum in to the backend. I think it has to be done in four stages: o move it into the backend and have it start/stop automatically o move the autovacuum configuration parameters into postgresql.conf o modify the code to use

Re: [HACKERS] Autovacuum in the backend

2005-06-14 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: One goal for 8.1 is to move /contrib/pg_autovacuum in to the backend. I think it has to be done in four stages: o move it into the backend and have it start/stop automatically o move the autovacuum configuration parameters into

Re: [HACKERS] Autovacuum in the backend

2005-06-14 Thread Qingqing Zhou
Bruce Momjian pgman@candle.pha.pa.us writes One goal for 8.1 is to move /contrib/pg_autovacuum in to the backend. I think it has to be done in four stages: o move it into the backend and have it start/stop automatically The start/stop routine is quite like Bgwriter. It requires PgStats to

Re: [HACKERS] Autovacuum in the backend

2005-06-14 Thread Qingqing Zhou
Tom Lane [EMAIL PROTECTED] writes A small problem here is that until you get at least to step 3 (backend-standard error handling), none of it is going to be acceptable to commit. So I don't entirely buy Bruce's notion of bite-size pieces of work. You can certainly work on it in that

Re: [HACKERS] Autovacuum in the backend

2005-06-14 Thread Neil Conway
Qingqing Zhou wrote: The start/stop routine is quite like Bgwriter. It requires PgStats to be turned on. Wasn't the plan to rewrite pg_autovacuum to use the FSM rather than the stats collector? -Neil ---(end of broadcast)--- TIP 3: if

Re: [HACKERS] Autovacuum in the backend

2005-06-14 Thread Qingqing Zhou
Neil Conway [EMAIL PROTECTED] writes Wasn't the plan to rewrite pg_autovacuum to use the FSM rather than the stats collector? I don't understand. Currently the basic logic of pg_autovacuum is to use the pg_stat_all_tables numbers like n_tup_upd, n_tup_del to determine if a relation need to