Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-22 Thread David Fetter
On Thu, Sep 22, 2005 at 03:52:21PM +1000, Gavin Sherry wrote: On Wed, 31 Aug 2005, Tom Lane wrote: BTW ... the original Berkeley papers on Postgres make frequent reference to a vacuum daemon, which seems to be essentially what we're trying to build with autovacuum. Does anyone know if

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-22 Thread Michael Glaesemann
On Sep 22, 2005, at 3:55 PM, David Fetter wrote: On Thu, Sep 22, 2005 at 03:52:21PM +1000, Gavin Sherry wrote: On Wed, 31 Aug 2005, Tom Lane wrote: Well, I was just poking around the executor and noticed this in ExecDelete(): /* * Note: Normally one would think that we have to

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-21 Thread Gavin Sherry
On Wed, 31 Aug 2005, Tom Lane wrote: BTW ... the original Berkeley papers on Postgres make frequent reference to a vacuum daemon, which seems to be essentially what we're trying to build with autovacuum. Does anyone know if the Berkeley implementation ever actually had auto vacuuming, or was

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-01 Thread Simon Riggs
On Wed, 2005-08-31 at 22:21 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Wed, 2005-08-31 at 19:24 -0400, Tom Lane wrote: If you don't remove any tuples, you don't scan the indexes anyway IIRC. No. Even if you remove *zero* tuples, an index is still scanned twice.

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-01 Thread Tom Lane
Josh Berkus josh@agliodbs.com writes: So, will per-table XID tracking allow us to avoid *ever* vacuuming some tables? If your definition of ever is less than a billion transactions, sure. (As Simon points out, with time-partitioned data sets that could often be arranged, so it's not a

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-01 Thread Alvaro Herrera
On Thu, Sep 01, 2005 at 04:21:58AM -, Andrew - Supernews wrote: On 2005-09-01, Alvaro Herrera [EMAIL PROTECTED] wrote: On Thu, Sep 01, 2005 at 01:57:02AM +0100, Simon Riggs wrote: If you're using autovacuum then the problem is already taken care of. autovacuum will respond only to

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-01 Thread Simon Riggs
On Thu, 2005-09-01 at 10:29 +0100, Simon Riggs wrote: On Wed, 2005-08-31 at 22:21 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Wed, 2005-08-31 at 19:24 -0400, Tom Lane wrote: If you don't remove any tuples, you don't scan the indexes anyway IIRC. No. Even if

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-09-01 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: My first proposal is to add an extra parameter onto the index_bulk_delete() call - ntuples. If ntuples == 0 then btbulkdelete() will avoid scanning and return immediately. If a scan occurs, then we keep track of how many tuples have been marked deleted and

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: A new command is proposed - VACUUM MINIMAL. The *sole* purpose of this command is to do the absolute minimum required to avoid transaction id wraparound. (Better names welcome) I do not see the point. If you only need to run it every billion

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Simon Riggs
On Wed, 2005-08-31 at 19:24 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: A new command is proposed - VACUUM MINIMAL. The *sole* purpose of this command is to do the absolute minimum required to avoid transaction id wraparound. (Better names welcome) I do not see the

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Alvaro Herrera
On Thu, Sep 01, 2005 at 01:57:02AM +0100, Simon Riggs wrote: If you're using autovacuum then the problem is already taken care of. autovacuum will respond only to UPDATEs and DELETEs. In the scenario I outline, these will *never* occur on the largest tables. A VACUUM would still eventually

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Wed, 2005-08-31 at 19:24 -0400, Tom Lane wrote: If you don't remove any tuples, you don't scan the indexes anyway IIRC. No. Even if you remove *zero* tuples, an index is still scanned twice. Once to not delete the rows and once to not delete the

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Christopher Kings-Lynne
I really really do not like proposals to introduce still another kind of VACUUM. We have too many already; any casual glance through the archives will show that most PG users don't have a grip on when to use VACUUM FULL vs VACUUM. Throwing in some more types will make that problem exponentially

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Tom Lane
Christopher Kings-Lynne [EMAIL PROTECTED] writes: I really really do not like proposals to introduce still another kind of VACUUM. We have too many already; any casual glance through the archives will show that most PG users don't have a grip on when to use VACUUM FULL vs VACUUM. Throwing in

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Gavin Sherry
On Wed, 31 Aug 2005, Tom Lane wrote: Christopher Kings-Lynne [EMAIL PROTECTED] writes: I really really do not like proposals to introduce still another kind of VACUUM. We have too many already; any casual glance through the archives will show that most PG users don't have a grip on when

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Josh Berkus
Gavin, Tom, Well, from my reading of some of the early papers, VACUUM was kind of different to what it is now. The idea was that expired data would be moved out the heap and stored else where. A timetravel mechanism could be used to see previous versions of the row. And from talking to a

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Josh Berkus
Tom, If you're using autovacuum then the problem is already taken care of. It will be taken care of better in 8.2, if we add per-table tracking of XID wraparound risk, but it's handled now.  The only way that this proposal makes any sense is if you are trying not to vacuum at all, ever. Hmmm

Re: [HACKERS] Minimally avoiding Transaction Wraparound in VLDBs

2005-08-31 Thread Andrew - Supernews
On 2005-09-01, Alvaro Herrera [EMAIL PROTECTED] wrote: On Thu, Sep 01, 2005 at 01:57:02AM +0100, Simon Riggs wrote: If you're using autovacuum then the problem is already taken care of. autovacuum will respond only to UPDATEs and DELETEs. In the scenario I outline, these will *never* occur