Re: [HACKERS] Command to prune archive at restartpoints

2010-06-14 Thread Dimitri Fontaine
Fujii Masao masao.fu...@gmail.com writes: In SR, WAL files in the pg_xlog directory on the standby are recycled by every restartpoints. So your proposed function seems not to be helpful even if hot_standby = on. Then I guess I'm at a loss here: what is the pg_archivecleanup utility good for in

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-14 Thread Simon Riggs
On Mon, 2010-06-14 at 12:21 +0200, Dimitri Fontaine wrote: Fujii Masao masao.fu...@gmail.com writes: In SR, WAL files in the pg_xlog directory on the standby are recycled by every restartpoints. So your proposed function seems not to be helpful even if hot_standby = on. Then I guess I'm

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-14 Thread Dimitri Fontaine
Simon Riggs si...@2ndquadrant.com writes: Cleaning the archive directory, not the pg_xlog directory. Hence the choice of the directory where to act. I was slow on that, sorry guys. I guess my main problem here is that I still picture PostgreSQL has being able to maintain an archive with no

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-13 Thread Robert Haas
On Thu, Jun 10, 2010 at 4:09 PM, Simon Riggs si...@2ndquadrant.com wrote: Here's the code. I haven't more than glanced at this, but +1 for committing it if you're confident it DTRT. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-13 Thread Robert Haas
On Sat, Jun 12, 2010 at 4:51 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: Dimitri Fontaine dfonta...@hi-media.com writes: Also, should I try to send a patch implementing my proposal (internal command exposed as a function at the SQL level, and while at it, maybe the internal command

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-13 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: I'm a bit perplexed here. The archive cleanup has to run on the standby, not the master, right? Whereas pg_switch_xlog() can only run on the master. I used it just to show a possible use case, easy to grasp. Sorry if that's confusing instead. The

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-13 Thread Robert Haas
On Sun, Jun 13, 2010 at 1:04 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: Robert Haas robertmh...@gmail.com writes: I'm a bit perplexed here.  The archive cleanup has to run on the standby, not the master, right?  Whereas pg_switch_xlog() can only run on the master. I used it just to

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-13 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: Robert Haas robertmh...@gmail.com writes:  The purpose of making this a standalone executable is so that people who have, for example, multiple standbys, can customize the logic without having to hack the backend.  Pushing this into the backend would

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-13 Thread Andrew Dunstan
Dimitri Fontaine wrote: I still read here from time to time that contrib's chapter is maintaining working examples of extensibility, not maintaining production ready add-ons. Even if this were true, and I don't believe it is, ISTM the solution would be to have a utility command

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-13 Thread Fujii Masao
On Mon, Jun 14, 2010 at 3:51 AM, Dimitri Fontaine dfonta...@hi-media.com wrote: I now realize that my proposal ain't helping if Streaming Replication is filling the standby's pg_xlog and hot_standby = off. I don't remember that SR rebuilds pg_xlog on the standby though, does it? In SR, WAL

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-12 Thread Dimitri Fontaine
Dimitri Fontaine dfonta...@hi-media.com writes: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: So to clean up all WAL files older than those needed by that base backup, you would simply copy-paste that location and call pg_cleanuparchive: pg_cleanuparchive /walarchive/

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-12 Thread Dimitri Fontaine
Dimitri Fontaine dfonta...@hi-media.com writes: Also, should I try to send a patch implementing my proposal (internal command exposed as a function at the SQL level, and while at it, maybe the internal command pg_archive_bypass to mimic /usr/bin/true as an archive_command)? I had to have a

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-11 Thread Dimitri Fontaine
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: So to clean up all WAL files older than those needed by that base backup, you would simply copy-paste that location and call pg_cleanuparchive: pg_cleanuparchive /walarchive/ 0001002F Ok, idle though: what about

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Heikki Linnakangas
On 09/06/10 10:21, Simon Riggs wrote: On Tue, 2010-06-08 at 18:30 -0400, Andrew Dunstan wrote: I prefer archive_cleanup_command. We should name things after their principal function, not an implementation detail, IMNSHO. More importantly, we should include an example in the docs. I created

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Simon Riggs
On Thu, 2010-06-10 at 10:18 +0300, Heikki Linnakangas wrote: On 09/06/10 10:21, Simon Riggs wrote: On Tue, 2010-06-08 at 18:30 -0400, Andrew Dunstan wrote: I prefer archive_cleanup_command. We should name things after their principal function, not an implementation detail, IMNSHO.

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Robert Haas
On Thu, Jun 10, 2010 at 3:28 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-06-10 at 10:18 +0300, Heikki Linnakangas wrote: On 09/06/10 10:21, Simon Riggs wrote: On Tue, 2010-06-08 at 18:30 -0400, Andrew Dunstan wrote: I prefer archive_cleanup_command. We should name things

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Andrew Dunstan
Robert Haas wrote: It won't kill us to change that sentence. pg_standby is only used now within the cleanup command etc pg_standby already contains the exact logic we need here. Having two sets of code for the same thing isn't how we do things. Well, we could factor out that part of

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Heikki Linnakangas
On 10/06/10 17:38, Andrew Dunstan wrote: I think my logic needs a tiny piece of adjustment, to ignore the timeline segment of the file name. I'm not sure you should ignore it. Presumably anything in an older timeline is indeed not required anymore and can be removed, and anything in a newer

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Andrew Dunstan
Heikki Linnakangas wrote: On 10/06/10 17:38, Andrew Dunstan wrote: I think my logic needs a tiny piece of adjustment, to ignore the timeline segment of the file name. I'm not sure you should ignore it. Presumably anything in an older timeline is indeed not required anymore and can be

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Heikki Linnakangas
On 10/06/10 22:24, Dimitri Fontaine wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes: Maybe we could add a new pg_cleanuparchive binary, but we'll need some discussion... Would this binary ever be used manually, not invoked by PostgreSQL? As it depends on the %r option to

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Dimitri Fontaine
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Maybe we could add a new pg_cleanuparchive binary, but we'll need some discussion... Would this binary ever be used manually, not invoked by PostgreSQL? As it depends on the %r option to be given and to be right, I don't think so.

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-10 Thread Simon Riggs
On Thu, 2010-06-10 at 22:49 +0300, Heikki Linnakangas wrote: On 10/06/10 22:24, Dimitri Fontaine wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes: Maybe we could add a new pg_cleanuparchive binary, but we'll need some discussion... Would this binary ever be used

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-09 Thread Simon Riggs
On Wed, 2010-06-09 at 11:18 +0900, Takahiro Itagaki wrote: Robert Haas robertmh...@gmail.com wrote: I think we're replacing restartpoint_command, not recovery_end_command. Ah, sorry. I did the same replacement for restartpoint_command in _, -, and camel case words. BTW, should we also

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-09 Thread Simon Riggs
On Tue, 2010-06-08 at 18:30 -0400, Andrew Dunstan wrote: I prefer archive_cleanup_command. We should name things after their principal function, not an implementation detail, IMNSHO. More importantly, we should include an example in the docs. I created one the other day when this was

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Robert Haas
On Mon, Mar 22, 2010 at 11:58 AM, Greg Stark gsst...@mit.edu wrote: On Thu, Mar 18, 2010 at 9:43 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-03-17 at 11:37 +0200, Heikki Linnakangas wrote: One awkward omission in the new built-in standby mode, mainly used for streaming

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Simon Riggs
On Tue, 2010-06-08 at 17:17 -0400, Robert Haas wrote: On Mon, Mar 22, 2010 at 11:58 AM, Greg Stark gsst...@mit.edu wrote: On Thu, Mar 18, 2010 at 9:43 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-03-17 at 11:37 +0200, Heikki Linnakangas wrote: One awkward omission in the new

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Andrew Dunstan
Robert Haas wrote: On Mon, Mar 22, 2010 at 11:58 AM, Greg Stark gsst...@mit.edu wrote: On Thu, Mar 18, 2010 at 9:43 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-03-17 at 11:37 +0200, Heikki Linnakangas wrote: One awkward omission in the new built-in standby

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: I prefer archive_cleanup_command. We should name things after their principal function, not an implementation detail, IMNSHO. Weak preference for archive_cleanup_command here. More importantly, we should include an example in the docs. I created

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Andrew Dunstan
Tom Lane wrote: As for the language choice, my first thought is +1 for perl over shell, mainly because it might be directly useful to people on Windows while shell never would be. On the other hand, if it's possible to do a useful one-liner in shell then let's do it that way. I don't

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Robert Haas
On Tue, Jun 8, 2010 at 6:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I prefer archive_cleanup_command. We should name things after their principal function, not an implementation detail, IMNSHO. Weak preference for archive_cleanup_command here. OK,

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Takahiro Itagaki
Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 8, 2010 at 6:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I prefer archive_cleanup_command. We should name things after their principal function, not an implementation detail, IMNSHO. Weak

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Robert Haas
On Tue, Jun 8, 2010 at 9:45 PM, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 8, 2010 at 6:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I prefer archive_cleanup_command. We should name

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Fujii Masao
On Wed, Jun 9, 2010 at 10:45 AM, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 8, 2010 at 6:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I prefer archive_cleanup_command. We should name

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Takahiro Itagaki
Robert Haas robertmh...@gmail.com wrote: I think we're replacing restartpoint_command, not recovery_end_command. Ah, sorry. I did the same replacement for restartpoint_command in _, -, and camel case words. BTW, should we also have a release note for the command? I added a simple description

Re: [HACKERS] Command to prune archive at restartpoints

2010-06-08 Thread Robert Haas
On Tue, Jun 8, 2010 at 10:18 PM, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: Robert Haas robertmh...@gmail.com wrote: I think we're replacing restartpoint_command, not recovery_end_command. Ah, sorry. I did the same replacement for restartpoint_command in _, -, and camel case

Re: [HACKERS] Command to prune archive at restartpoints

2010-03-22 Thread Greg Stark
On Thu, Mar 18, 2010 at 9:43 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-03-17 at 11:37 +0200, Heikki Linnakangas wrote: One awkward omission in the new built-in standby mode, mainly used for streaming replication, is that there is no easy way to delete old archived files like

Re: [HACKERS] Command to prune archive at restartpoints

2010-03-18 Thread Heikki Linnakangas
Committed. Heikki Linnakangas wrote: One awkward omission in the new built-in standby mode, mainly used for streaming replication, is that there is no easy way to delete old archived files like you do with the %r parameter to restore_command. This was discussed at

Re: [HACKERS] Command to prune archive at restartpoints

2010-03-18 Thread Simon Riggs
On Wed, 2010-03-17 at 11:37 +0200, Heikki Linnakangas wrote: One awkward omission in the new built-in standby mode, mainly used for streaming replication, is that there is no easy way to delete old archived files like you do with the %r parameter to restore_command. This was discussed at

[HACKERS] Command to prune archive at restartpoints

2010-03-17 Thread Heikki Linnakangas
One awkward omission in the new built-in standby mode, mainly used for streaming replication, is that there is no easy way to delete old archived files like you do with the %r parameter to restore_command. This was discussed at http://archives.postgresql.org/pgsql-hackers/2010-02/msg01003.php,

Re: [HACKERS] Command to prune archive at restartpoints

2010-03-17 Thread Greg Stark
On Wed, Mar 17, 2010 at 9:37 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: One awkward omission in the new built-in standby mode, mainly used for streaming replication, is that there is no easy way to delete old archived files like you do with the %r parameter to

Re: [HACKERS] Command to prune archive at restartpoints

2010-03-17 Thread Heikki Linnakangas
Greg Stark wrote: On Wed, Mar 17, 2010 at 9:37 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: One awkward omission in the new built-in standby mode, mainly used for streaming replication, is that there is no easy way to delete old archived files like you do with the %r