Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-29 Thread Robert Haas
On Tue, Apr 28, 2015 at 2:44 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think what we need here is something that does heap_update to tuples at the end of the table, moving them to earlier pages; then wait for old snapshots to die (the infrastructure for which we have now, thanks

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-29 Thread Jeff Janes
On Tue, Apr 28, 2015 at 11:32 AM, Robert Haas robertmh...@gmail.com wrote: On Fri, Apr 24, 2015 at 3:04 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think what we need here is something that does heap_update to tuples at the end of the table, moving them to earlier pages; then wait

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-28 Thread Jim Nasby
On 4/28/15 1:32 PM, Robert Haas wrote: More than five years have passed since Heikki posted this, and we still haven't found a solution to the problem -- which neverthless keeps biting people to the point that multiple user-space implementations of similar techniques are out there. Yeah. The

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-28 Thread Robert Haas
On Fri, Apr 24, 2015 at 3:04 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Heikki Linnakangas wrote: Hmm, AFAICT pg_reorg is much more complex, writing stuff to a temp table and swapping relfilenodes afterwards. More like the VACUUM REWRITE that's been discussed. For the kicks, I looked

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-28 Thread Alvaro Herrera
Robert Haas wrote: On Fri, Apr 24, 2015 at 3:04 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I think what we need here is something that does heap_update to tuples at the end of the table, moving them to earlier pages; then wait for old snapshots to die (the infrastructure for which

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-27 Thread Jim Nasby
On 4/25/15 6:30 AM, Simon Riggs wrote: On 24 April 2015 at 22:36, Jim Nasby jim.na...@bluetreble.com mailto:jim.na...@bluetreble.com wrote: Instead of adding forcefsm, I think it would be more useful to accept a target block number. That way we can actually control where the new

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-25 Thread Simon Riggs
On 24 April 2015 at 22:36, Jim Nasby jim.na...@bluetreble.com wrote: Instead of adding forcefsm, I think it would be more useful to accept a target block number. That way we can actually control where the new tuple goes. For this particular case we'd presumably go with normal FSM page

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-24 Thread Alvaro Herrera
Jim Nasby wrote: It looks like the biggest complaint (aside from allowing a limited number of tuples to be moved) is in [1] and [2], where Tom is saying that you can't simply call heap_update() like this without holding an exclusive lock on the table. Is that because we're not actually

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-24 Thread Jim Nasby
On 4/24/15 5:30 PM, Alvaro Herrera wrote: Jim Nasby wrote: It looks like the biggest complaint (aside from allowing a limited number of tuples to be moved) is in [1] and [2], where Tom is saying that you can't simply call heap_update() like this without holding an exclusive lock on the table.

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-24 Thread Alvaro Herrera
Heikki Linnakangas wrote: Hmm, AFAICT pg_reorg is much more complex, writing stuff to a temp table and swapping relfilenodes afterwards. More like the VACUUM REWRITE that's been discussed. For the kicks, I looked at what it would take to write a utility like that. It turns out to be quite

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-24 Thread Jim Nasby
On 4/24/15 2:04 PM, Alvaro Herrera wrote: Heikki Linnakangas wrote: Hmm, AFAICT pg_reorg is much more complex, writing stuff to a temp table and swapping relfilenodes afterwards. More like the VACUUM REWRITE that's been discussed. For the kicks, I looked at what it would take to write a

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-24 Thread Alvaro Herrera
Jim Nasby wrote: Honestly, I'd prefer we exposed some way to influence where a new tuple gets put, and perhaps better ways of accessing tuples on a specific page. That would make it a lot easier to handle this in userspace, but it would also make it easier to do things like concurrent

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2015-04-24 Thread Jim Nasby
On 4/24/15 3:34 PM, Alvaro Herrera wrote: Jim Nasby wrote: Honestly, I'd prefer we exposed some way to influence where a new tuple gets put, and perhaps better ways of accessing tuples on a specific page. That would make it a lot easier to handle this in userspace, but it would also make it

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-18 Thread Albe Laurenz
Tom Lane wrote: I don't see any reason why not breaking the user visible behavior of tuples CTID between any two major releases, Am I completely wet here? Completely. This is a user-visible behavior that we have encouraged people to rely on, and for which there is no easy substitute.

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-18 Thread Simon Riggs
On Thu, 2009-09-17 at 17:44 -0400, Tom Lane wrote: Dimitri Fontaine dfonta...@hi-media.com writes: I don't see any reason why not breaking the user visible behavior of tuples CTID between any two major releases, Am I completely wet here? Completely. This is a user-visible behavior

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-18 Thread Simon Riggs
On Fri, 2009-09-18 at 08:50 +0200, Albe Laurenz wrote: Tom Lane wrote: I don't see any reason why not breaking the user visible behavior of tuples CTID between any two major releases, Am I completely wet here? Completely. This is a user-visible behavior that we have encouraged

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-18 Thread marcin mank
Exactly.  The application is typically going to throw a concurrent update type of error when this happens, and we don't want magic background operations to cause that. I`d give up the possibility of using CTIDs in the way You explained for an auto-debloater without blinking an eye. Maybe we

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-18 Thread Heikki Linnakangas
Simon Riggs wrote: CTIDs don't help with optimistic locking, though it seems they can. If you don't hold open the transaction then someone else can update the row. That sounds good, but because of HOT it is possible that the same CTID with the same PK value occupies that exact CTID value

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Simon Riggs
On Wed, 2009-09-16 at 23:12 -0400, Tom Lane wrote: Jeff Davis pg...@j-davis.com writes: On Wed, 2009-09-16 at 21:48 -0400, Tom Lane wrote: Yeah, I was just wondering about that myself. Seems like there would be lots of situations where short exclusive-lock intervals could be tolerated,

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Hannu Krosing
On Wed, 2009-09-16 at 21:19 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: What we need is VACUUM FULL CONCURRENTLY and REINDEX CONCURRENTLY. VACUUM FULL CONCURRENTLY is a contradiction in terms. Wishing it were possible doesn't make it so. It depends on what do you mean

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Joshua Tolley
On Wed, Sep 16, 2009 at 09:48:20PM -0400, Tom Lane wrote: Seems like there would be lots of situations where short exclusive-lock intervals could be tolerated, even though not long ones. So that's another argument for being able to set an upper bound on how many tuples get moved per call.

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Robert Haas
On Thu, Sep 17, 2009 at 9:35 AM, Joshua Tolley eggyk...@gmail.com wrote: On Wed, Sep 16, 2009 at 09:48:20PM -0400, Tom Lane wrote: Seems like there would be lots of situations where short exclusive-lock intervals could be tolerated, even though not long ones.  So that's another argument for

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2009-09-16 at 21:19 -0400, Tom Lane wrote: VACUUM FULL CONCURRENTLY is a contradiction in terms. Wishing it were possible doesn't make it so. It depends on what do you mean by VACUUM FULL Anything that moves tuples is not acceptable as a

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Hannu Krosing
On Thu, 2009-09-17 at 09:45 -0400, Robert Haas wrote: On Thu, Sep 17, 2009 at 9:35 AM, Joshua Tolley eggyk...@gmail.com wrote: On Wed, Sep 16, 2009 at 09:48:20PM -0400, Tom Lane wrote: Seems like there would be lots of situations where short exclusive-lock intervals could be tolerated,

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Hannu Krosing
On Thu, 2009-09-17 at 10:21 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2009-09-16 at 21:19 -0400, Tom Lane wrote: VACUUM FULL CONCURRENTLY is a contradiction in terms. Wishing it were possible doesn't make it so. It depends on what do you mean by VACUUM

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes: On Thu, 2009-09-17 at 10:21 -0400, Tom Lane wrote: because it will break applications that depend on CTID. Do you know of any such applications out in the wild ? Yes, they're out there. regards, tom lane -- Sent via

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes: On Thu, 2009-09-17 at 09:45 -0400, Robert Haas wrote: Making the code more complicated so that it's easier to tune something that isn't very hard to tune anyway doesn't seem like a good trade-off. I think that just making sure that pessimal cases

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Robert Haas
On Thu, Sep 17, 2009 at 10:21 AM, Tom Lane t...@sss.pgh.pa.us wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2009-09-16 at 21:19 -0400, Tom Lane wrote: VACUUM FULL CONCURRENTLY is a contradiction in terms.  Wishing it were possible doesn't make it so. It depends on what do you

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Simon Riggs
On Thu, 2009-09-17 at 10:45 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Thu, 2009-09-17 at 09:45 -0400, Robert Haas wrote: Making the code more complicated so that it's easier to tune something that isn't very hard to tune anyway doesn't seem like a good

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Simon Riggs
On Thu, 2009-09-17 at 11:25 -0400, Robert Haas wrote: On Thu, Sep 17, 2009 at 10:21 AM, Tom Lane t...@sss.pgh.pa.us wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2009-09-16 at 21:19 -0400, Tom Lane wrote: VACUUM FULL CONCURRENTLY is a contradiction in terms. Wishing it were

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Heikki Linnakangas
Robert Haas wrote: On Thu, Sep 17, 2009 at 10:21 AM, Tom Lane t...@sss.pgh.pa.us wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Wed, 2009-09-16 at 21:19 -0400, Tom Lane wrote: VACUUM FULL CONCURRENTLY is a contradiction in terms. Wishing it were possible doesn't make it so. It

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Robert Haas wrote: On Thu, Sep 17, 2009 at 10:21 AM, Tom Lane t...@sss.pgh.pa.us wrote: Anything that moves tuples is not acceptable as a hidden background operation, because it will break applications that depend on CTID. I'm a

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Hannu Krosing
On Thu, 2009-09-17 at 10:32 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Thu, 2009-09-17 at 10:21 -0400, Tom Lane wrote: because it will break applications that depend on CTID. Do you know of any such applications out in the wild ? Yes, they're out there. How

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: The update utility being discussed is in danger of confusing these two goals * compact the table using minimal workspace * compact the table with minimal interruption to concurrent updaters Actually, the update utility is explicitly meant to satisfy

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Hannu Krosing
On Thu, 2009-09-17 at 12:18 -0400, Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Robert Haas wrote: On Thu, Sep 17, 2009 at 10:21 AM, Tom Lane t...@sss.pgh.pa.us wrote: Anything that moves tuples is not acceptable as a hidden background operation, because

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Robert Haas
On Thu, Sep 17, 2009 at 12:18 PM, Tom Lane t...@sss.pgh.pa.us wrote: It's no different from the situation where another backend UPDATEs the row under your nose, but it's not something you want to do automatically without notice. Exactly.  The application is typically going to throw a

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Robert Haas
On Thu, Sep 17, 2009 at 12:31 PM, Hannu Krosing ha...@2ndquadrant.com wrote: Exactly.  The application is typically going to throw a concurrent update type of error when this happens, and we don't want magic background operations to cause that. Would'nt current VACUUM FULL or CLUSTER cause

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes: On Thu, 2009-09-17 at 12:18 -0400, Tom Lane wrote: Or for an update without having to hold a transaction open. We have recommended this type of technique in the past: If you need real locking, then just define a locked (or locked_by or

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Simon Riggs
On Thu, 2009-09-17 at 12:30 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: The update utility being discussed is in danger of confusing these two goals * compact the table using minimal workspace * compact the table with minimal interruption to concurrent updaters

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Greg Smith
On Wed, 16 Sep 2009, Tom Lane wrote: * Shrink a table in place - when no space available To be addressed by the UPDATE-style tuple-mover (which could be thought of as VACUUM FULL rewritten to not use any special mechanisms). Is there any synergy here with the needs of a future in-place

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Hannu Krosing
On Thu, 2009-09-17 at 12:36 -0400, Tom Lane wrote: Hannu Krosing ha...@2ndquadrant.com writes: On Thu, 2009-09-17 at 12:18 -0400, Tom Lane wrote: Or for an update without having to hold a transaction open. We have recommended this type of technique in the past: If you need real

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Hannu Krosing
On Thu, 2009-09-17 at 14:33 -0400, Greg Smith wrote: On Wed, 16 Sep 2009, Tom Lane wrote: * Shrink a table in place - when no space available To be addressed by the UPDATE-style tuple-mover (which could be thought of as VACUUM FULL rewritten to not use any special mechanisms). Is there

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Dimitri Fontaine
Hi, Forewords: re-reading, I hope my english will not make this sound like a high-kick when I'm just struggling to understand what all this is about. Sending in order not to regret missing the oportunity I think I'm seeing... Tom Lane t...@sss.pgh.pa.us writes: Hannu Krosing

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Hannu Krosing ha...@2ndquadrant.com writes: But maybe something can be tahen from this discussion the other way round - maybe we should not be afraid of doing null updates during in-place update The problem for in-place update is that it can't assume that any of the normal infrastructure (like

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Tom Lane
Dimitri Fontaine dfonta...@hi-media.com writes: I don't see any reason why not breaking the user visible behavior of tuples CTID between any two major releases, Am I completely wet here? Completely. This is a user-visible behavior that we have encouraged people to rely on, and for which

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-17 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: Completely. This is a user-visible behavior that we have encouraged people to rely on, and for which there is no easy substitute. Excited to have self-healing tables (against bloat), I parse this as an opening. Previously on this thread you say: (Actually,

[HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Josh Berkus
Hackers, Here's the feedback on replacing VACUUM FULL with VACUUM REWRITE: http://it.toolbox.com/blogs/database-soup/getting-rid-of-vacuum-full-feedback-needed-33959 Of note: a) To date, I have yet to hear a single person bring up an actual real-life use-case where VACUUM FULL was desireable

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Kevin Grittner
Josh Berkus j...@agliodbs.com wrote: a) To date, I have yet to hear a single person bring up an actual real-life use-case where VACUUM FULL was desireable and REWRITE would not be. Would rewrite have handled this?: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01016.php -Kevin

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Heikki Linnakangas
Josh Berkus wrote: Hackers, Here's the feedback on replacing VACUUM FULL with VACUUM REWRITE: http://it.toolbox.com/blogs/database-soup/getting-rid-of-vacuum-full-feedback-needed-33959 Of note: a) To date, I have yet to hear a single person bring up an actual real-life use-case where

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Jeff Davis
On Wed, 2009-09-16 at 21:23 +0300, Heikki Linnakangas wrote: 2) Another utility that does something like UPDATE ... WHERE ctid ? to move tuples to lower pages. It will be different from current VACUUM FULL in some ways. It won't require a table lock, for example, but it won't be able to move

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Josh Berkus
On 9/16/09 11:20 AM, Kevin Grittner wrote: Josh Berkus j...@agliodbs.com wrote: a) To date, I have yet to hear a single person bring up an actual real-life use-case where VACUUM FULL was desireable and REWRITE would not be. Would rewrite have handled this?:

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Hannu Krosing
On Wed, 2009-09-16 at 11:10 -0700, Josh Berkus wrote: Hackers, Here's the feedback on replacing VACUUM FULL with VACUUM REWRITE: http://it.toolbox.com/blogs/database-soup/getting-rid-of-vacuum-full-feedback-needed-33959 Of note: a) To date, I have yet to hear a single person bring up an

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Hannu Krosing
On Wed, 2009-09-16 at 21:23 +0300, Heikki Linnakangas wrote: I was just going to post that we should make a decision about this, because ISTM there's some code in Simon's hot standby patch that is only required to support VACUUM FULL. If we make the decision that we drop VACUUM FULL in 8.5,

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Josh Berkus
Hannu, The only case is when you are out of disk space and can't afford to write out a full set of live rows. Well, it's actually rather specific. You need to have: a) *Some* free disk space (FULL requires extra disk) but not enough to copy one entire table and its indexes. b) be already

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Hannu Krosing
On Wed, 2009-09-16 at 11:10 -0700, Josh Berkus wrote: Hackers, Here's the feedback on replacing VACUUM FULL with VACUUM REWRITE: http://it.toolbox.com/blogs/database-soup/getting-rid-of-vacuum-full-feedback-needed-33959 Of note: a) To date, I have yet to hear a single person bring up an

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Hannu Krosing
On Wed, 2009-09-16 at 13:20 -0500, Kevin Grittner wrote: Josh Berkus j...@agliodbs.com wrote: a) To date, I have yet to hear a single person bring up an actual real-life use-case where VACUUM FULL was desireable and REWRITE would not be. Would rewrite have handled this?:

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Hannu Krosing
On Wed, 2009-09-16 at 11:48 -0700, Josh Berkus wrote: Hannu, The only case is when you are out of disk space and can't afford to write out a full set of live rows. Well, it's actually rather specific. You need to have: a) *Some* free disk space (FULL requires extra disk) but not

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Josh Berkus
Hannu, If it is a sequence of 1. ordinary VACUUM (it can't run out of FSM anymore, no?) 2. a process moving live tuples from end (using reverse seqscan) to free space found scanning in first-to-last direction, either one tuple at a time or one page at a time, until the two scans meet 3.

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Heikki Linnakangas
Hannu Krosing wrote: On Wed, 2009-09-16 at 21:23 +0300, Heikki Linnakangas wrote: 2) Another utility that does something like UPDATE ... WHERE ctid ? to move tuples to lower pages. It will be different from current VACUUM FULL in some ways. It won't require a table lock, for example, but it

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Hannu Krosing
On Wed, 2009-09-16 at 23:53 +0300, Heikki Linnakangas wrote: Hannu Krosing wrote: On Wed, 2009-09-16 at 21:23 +0300, Heikki Linnakangas wrote: 2) Another utility that does something like UPDATE ... WHERE ctid ? to move tuples to lower pages. It will be different from current VACUUM FULL

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Jeff Davis
On Wed, 2009-09-16 at 23:53 +0300, Heikki Linnakangas wrote: For the kicks, I looked at what it would take to write a utility like that. It turns out to be quite trivial, patch attached. It uses the same principle as VACUUM FULL, scans from the end, moving tuples to lower-numbered pages until

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Robert Haas
On Wed, Sep 16, 2009 at 4:53 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Hannu Krosing wrote: On Wed, 2009-09-16 at 21:23 +0300, Heikki Linnakangas wrote: 2) Another utility that does something like UPDATE ... WHERE ctid ? to move tuples to lower pages. It will be

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Jaime Casanova
On Wed, Sep 16, 2009 at 1:42 PM, Josh Berkus j...@agliodbs.com wrote: On 9/16/09 11:20 AM, Kevin Grittner wrote: Josh Berkus j...@agliodbs.com wrote: a) To date, I have yet to hear a single person bring up an actual real-life use-case where VACUUM FULL was desireable and REWRITE would not

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: I think this should be in core, not a contrib module. +1 I also wonder whether we should consider teaching regular VACUUM to do a little of this every time it's run. Right now, once your table gets bloated, it stays bloated forever, until you

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Ron Mayer
Robert Haas wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Hannu Krosing wrote: On Wed, 2009-09-16 at 21:23 +0300, Heikki Linnakangas wrote: 2) Another utility that does something like UPDATE ... WHERE ctid ? to I also wonder whether we should consider teaching regular

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: For the kicks, I looked at what it would take to write a utility like that. It turns out to be quite trivial, patch attached. I don't think you've really thought this through; particularly not this: + rel = heap_open(relid,

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I'm in favor of removing VACUUM FULL in 8.5. To replace it, we should offer: 1) VACUUM REWRITE, which is like CLUSTER but doesn't use an index, and Check, although I'm not eager to make REWRITE a fully reserved word, which is what

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Simon Riggs
On Wed, 2009-09-16 at 21:23 +0300, Heikki Linnakangas wrote: I was just going to post that we should make a decision about this, because ISTM there's some code in Simon's hot standby patch that is only required to support VACUUM FULL. If we make the decision that we drop VACUUM FULL in 8.5,

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Simon Riggs
On Wed, 2009-09-16 at 11:40 -0700, Jeff Davis wrote: Another thing to think about is that lazy vacuum only shrinks the heap file if it happens to be able to acquire an access exclusive lock. Because vacuum can't be run inside a transaction block, I don't think there's currently a way to

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Andrew McNamara
Well, Andrew McNamara just posted today: http://archives.postgresql.org/message-id/20090916063341.0735c5ac...@longblack.object-craft.com.au Had VACUUM FULL not been available, though, I'm pretty sure he would've come up with something else instead. Indeed I would have. And it was our own

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Simon Riggs
On Wed, 2009-09-16 at 20:36 -0400, Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: For the kicks, I looked at what it would take to write a utility like that. It turns out to be quite trivial, patch attached. I don't think you've really thought this through;

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: What we need is VACUUM FULL CONCURRENTLY and REINDEX CONCURRENTLY. VACUUM FULL CONCURRENTLY is a contradiction in terms. Wishing it were possible doesn't make it so. regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Simon Riggs
On Wed, 2009-09-16 at 21:00 -0400, Tom Lane wrote: But if we get rid of old-style VACUUM FULL then we do need something to cover those few-and-far-between situations where you really do desperately need to compact a table in place; and a utility like this seems like a reasonable solution.

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: I think it would be useful to have an additional option to force VACUUM to wait for the lock so it can truncate. It's annoying to have to re-run VACUUM just to give it a chance at the lock again. It would be better to separate out the

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: The way I read the thread so far is that there are multiple requirements: * Shrink a table efficiently - when time and space available to do so To be addressed by the CLUSTER-based solution (VACUUM REWRITE or whatever we call it). * Shrink a table

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Robert Haas
On Wed, Sep 16, 2009 at 9:35 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: The way I read the thread so far is that there are multiple requirements: * Shrink a table efficiently - when time and space available to do so To be addressed by the CLUSTER-based

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Wed, Sep 16, 2009 at 9:35 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: * Shrink a table concurrently - when no dedicated time available Wishful thinking, which should not stop us from proceeding with the

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Jeff Davis
On Wed, 2009-09-16 at 21:48 -0400, Tom Lane wrote: Yeah, I was just wondering about that myself. Seems like there would be lots of situations where short exclusive-lock intervals could be tolerated, even though not long ones. But a short-lived exclusive lock can turn into a long-lived

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: On Wed, 2009-09-16 at 21:48 -0400, Tom Lane wrote: Yeah, I was just wondering about that myself. Seems like there would be lots of situations where short exclusive-lock intervals could be tolerated, even though not long ones. But a short-lived exclusive

Re: [HACKERS] Feedback on getting rid of VACUUM FULL

2009-09-16 Thread Heikki Linnakangas
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Sep 16, 2009 at 9:35 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: * Shrink a table concurrently - when no dedicated time available Wishful thinking, which should not stop us from proceeding

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-24 Thread Peter Eisentraut
On fre, 2009-08-21 at 20:07 -0400, Tom Lane wrote: As of SQL99 it's supposed to be legal if you're grouping by a primary key (or some other cases where the other columns can be proved functionally dependent on the grouping columns, but that's the most useful one). We haven't got round to

[HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Jean-Michel Pouré
Dear friends, I have been using PostgreSQL since 6.3 releases and I am a real fan. Of course, I never use nor trust MySQL to deliver data. Now I use Drupal 6.3 with PostgreSQL 8.4. I loose a lot of time correcting Drupal SQL. You may be interested in my developer feedback. I gathered some

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Joshua D. Drake
This page gathers most frequent problems that Drupal users and developers encounter when using PostgreSQL. I would be delighted to have your feedback. Could some issues reasonably be fixed for a better Drupal support? Well I doubt we would do anything to copy MySQL. However Drupal has

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread D'Arcy J.M. Cain
On Fri, 21 Aug 2009 18:22:41 +0200 Jean-Michel Pouré j...@poure.com wrote: I gathered some real examples here: Guidelines for writing MySQL and PostgreSQL compliant SQL = http://drupal.org/node/14 This page gathers most frequent problems that Drupal users and developers encounter when

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Bruce Momjian
Jean-Michel Pour? wrote: -- Start of PGP signed section. Dear friends, I have been using PostgreSQL since 6.3 releases and I am a real fan. Of course, I never use nor trust MySQL to deliver data. Now I use Drupal 6.3 with PostgreSQL 8.4. I loose a lot of time correcting Drupal SQL.

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Joshua D. Drake
On Fri, 2009-08-21 at 12:50 -0400, Bruce Momjian wrote: I show multi-value INSERT was added in PG 8.2, not 8.4, * Add support for multiple-row VALUES clauses, per SQL standard (Joe, Tom), http://drupal.org/node/68. I am confused because I thought Drupal worked with Postgres, but

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Andrew Dunstan
Joshua D. Drake wrote: This page gathers most frequent problems that Drupal users and developers encounter when using PostgreSQL. I would be delighted to have your feedback. Could some issues reasonably be fixed for a better Drupal support? Well I doubt we would do anything to copy

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Greg Stark
2009/8/21 Andrew Dunstan and...@dunslane.net: Since you haven't shown us what page this refers to, I at least am totally in the dark about what is being discussed. It was in the original post http://drupal.org/node/14 -- greg http://mit.edu/~gsstark/resume.pdf -- Sent via

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Andrew Dunstan
Greg Stark wrote: 2009/8/21 Andrew Dunstan and...@dunslane.net: Since you haven't shown us what page this refers to, I at least am totally in the dark about what is being discussed. It was in the original post http://drupal.org/node/14 Darn. Our mail system sucks badly.

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Robert Haas
2009/8/21 Jean-Michel Pouré j...@poure.com: Dear friends, I have been using PostgreSQL since 6.3 releases and I am a real fan. Of course, I never use nor trust MySQL to deliver data. Now I use Drupal 6.3 with PostgreSQL 8.4. I loose a lot of time correcting Drupal SQL. You may be

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Josh Berkus
Jean-Michel, Thank you for doing this! I've registered for the Drupal site so that I can fix and/or expand some of your items. People who know Drupal better than me should add to them. If you want to discuss Drupal PostgreSQL again, please post on the pgsql-advocacy list or the pgsql-php

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: BTW, why don't we have a multi-argument version of CONCAT()? Why wouldn't people use the SQL-standard || operator instead? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Jean-Michel Pouré
I've registered for the Drupal site so that I can fix and/or expand some of your items. Thanks. I corrected the index on dual fields page. If you want to discuss Drupal PostgreSQL again, please post on the pgsql-advocacy list or the pgsql-php mailing lists. pgsql-hackers isn't the best

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread David Fetter
On Fri, Aug 21, 2009 at 04:19:43PM -0400, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: BTW, why don't we have a multi-argument version of CONCAT()? Why wouldn't people use the SQL-standard || operator instead? Because by default, MySQL uses that as, get this, logical OR. Cheers,

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Tom Lane
David Fetter da...@fetter.org writes: On Fri, Aug 21, 2009 at 04:19:43PM -0400, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: BTW, why don't we have a multi-argument version of CONCAT()? Why wouldn't people use the SQL-standard || operator instead? Because by default, MySQL uses

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Josh Berkus
On 8/21/09 3:17 PM, Tom Lane wrote: David Fetter da...@fetter.org writes: On Fri, Aug 21, 2009 at 04:19:43PM -0400, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: BTW, why don't we have a multi-argument version of CONCAT()? Why wouldn't people use the SQL-standard || operator instead?

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Andrew Dunstan
Jean-Michel Pouré wrote: BTW, why don't we have a multi-argument version of CONCAT()? In 8.4, it would be possible ... I should add it to mysqlcompat library. yes. In PostgreSQL core ... No. That is exactly where it shouldn't go. And frankly, Drupal developers should stop

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Greg Stark
2009/8/21 Jean-Michel Pouré j...@poure.com: PostgreSQL requires all non-aggregated fields to be present in the GROUP BY clause (I fixed 10 such issues in Drupal code). http://drupal.org/node/30 Why can't PostgreSQL add the required field automatically? Could this be added to PostgreSQL

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Stephen Frost
* Greg Stark (gsst...@mit.edu) wrote: select distinct on (a) a,b,c from a ORDER BY a,b,c But Postgres insists you have an ORDER BY which has to agree with the DISTINCT ON columns and provide some extra column(s) to determine which values of b,c are chosen. Not quite technically correct.

Re: [HACKERS] Feedback about Drupal SQL debugging

2009-08-21 Thread Greg Stark
2009/8/22 Stephen Frost sfr...@snowman.net: Hrmm.  That sounds kinda neat, but you'd still have to specify one of the columns in the GROUP BY, I presume?  Or could you just say 'GROUP BY' without any columns, and have it GROUP BY the key of the table you're using? You would have to specify

  1   2   >