Re: [HACKERS] [COMMITTERS] pgsql: Add recovery_end_command option to recovery.conf.

2009-05-15 Thread Heikki Linnakangas
Tom Lane wrote: Alvaro Herrera alvhe...@commandprompt.com writes: Heikki Linnakangas wrote: Log Message: --- Add recovery_end_command option to recovery.conf. recovery_end_command is run at the end of archive recovery, providing a chance to do external cleanup. Modify pg_standby so

Re: [HACKERS] Optimizing Read-Only Scalability

2009-05-15 Thread Simon Riggs
On Thu, 2009-05-14 at 16:21 -0700, Josh Berkus wrote: So we can optimize away the scan through the procarray by doing two if tests, one outside of the lock, one inside. In normal running, both will be optimized away, though in read-only periods we would avoid much work. How much work

Re: [HACKERS] New trigger option of pg_standby

2009-05-15 Thread Simon Riggs
On Thu, 2009-05-14 at 23:31 +0300, Heikki Linnakangas wrote: I've finally committed changes to pg_standby. That was a good team effort. Thanks for committing. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 8, 2009 at 2:42 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: When you create a new base backup, you shouldn't need any files archived before starting the backup. If so, this fix is not enough, since findNewestTimeLine() is still based on the premise that

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Fujii Masao wrote: Hi, On Fri, May 8, 2009 at 2:42 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: When you create a new base backup, you shouldn't need any files archived before starting the backup. If so, this fix is not enough, since findNewestTimeLine() is still based

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 20:11 +0900, Fujii Masao wrote: Hi, On Fri, May 8, 2009 at 2:42 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: When you create a new base backup, you shouldn't need any files archived before starting the backup. If so, this fix is not enough,

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The probe in findNewestTimeLine() initialized to recovery target timeline + 1. It doesn't require history files for any old timelines to be present. What if recovery_target_timeline = 'latest'?

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 20:38 +0900, Fujii Masao wrote: On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The probe in findNewestTimeLine() initialized to recovery target timeline + 1. It doesn't require history files for any old timelines to be

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 12:56 +0100, Simon Riggs wrote: There is no particular reason to send history files to the archive, since new ones are only ever generated at the end of an archive recovery. It also clears up a long standing confusion between backup history files and timeline history

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Fujii Masao wrote: On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The probe in findNewestTimeLine() initialized to recovery target timeline + 1. It doesn't require history files for any old timelines to be present. What if

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 20:38 +0900, Fujii Masao wrote: On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The probe in findNewestTimeLine() initialized to recovery target timeline + 1. It doesn't require history files for any old

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 14:56 +0300, Heikki Linnakangas wrote: Simon's idea of keeping a copy of all the history files in the data directory wouldn't help here. In fact, I think we already never delete history files in the server, it's just that if you omit the pg_xlog directory in the base

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Mikael Krantz wrote: On Fri, May 15, 2009 at 2:22 PM, Simon Riggs si...@2ndquadrant.com wrote: Let's document that timeline files should not be deleted from the archive iff there exists a base backup made during a lower numbered timeline. Or made during a higher numbered timeline which

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Mikael Krantz
On Fri, May 15, 2009 at 2:22 PM, Simon Riggs si...@2ndquadrant.com wrote: Let's document that timeline files should not be deleted from the archive iff there exists a base backup made during a lower numbered timeline. Or made during a higher numbered timeline which happens to start in a

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: ehem, It will work fine isn't correct, as Fujii-san observes. What exactly are the steps required to run into that problem? I fail to see what the problem is. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 12:56 +0100, Simon Riggs wrote: There is no particular reason to send history files to the archive, since new ones are only ever generated at the end of an archive recovery. It also clears up a long standing confusion between backup history files and

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 15, 2009 at 9:22 PM, Simon Riggs si...@2ndquadrant.com wrote: If you delete history file and all the WAL for timeline 6, yeah, nothing stops it from being reused. It will work just fine, as if it never existed. If you still have the history file and WAL for the old timeline 6

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 15:41 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Fri, 2009-05-15 at 12:56 +0100, Simon Riggs wrote: There is no particular reason to send history files to the archive, since new ones are only ever generated at the end of an archive recovery. It

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Mikael Krantz
On Fri, May 15, 2009 at 2:26 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That was the original issue you ran into. That has now been fixed by forcing an xlog switch at pg_start_backup(), so that you can't start a backup in a WAL file that contains records from a lower

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 15:34 +0200, Mikael Krantz wrote: On Fri, May 15, 2009 at 2:26 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That was the original issue you ran into. That has now been fixed by forcing an xlog switch at pg_start_backup(), so that you can't start a

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 15, 2009 at 8:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: The probe in findNewestTimeLine() initialized to recovery target timeline +

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 22:56 +0900, Fujii Masao wrote: OK, I probably understood your point. The timeline history files whose timeline ID is larger than that of an oldest backup must not be deleted from the archive. On the other hand, the smaller or equal one can be deleted. Not all history

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Andrew Dunstan
Simon Riggs wrote: On Fri, 2009-05-15 at 22:56 +0900, Fujii Masao wrote: OK, I probably understood your point. The timeline history files whose timeline ID is larger than that of an oldest backup must not be deleted from the archive. On the other hand, the smaller or equal one can be

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 22:56 +0900, Fujii Masao wrote: OK, I probably understood your point. The timeline history files whose timeline ID is larger than that of an oldest backup must not be deleted from the archive. On the other hand, the smaller or equal one can be deleted.

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 17:19 +0300, Heikki Linnakangas wrote: Yes, just as deleting old WAL files. So what you're saying is because it's possible to blow your left foot off, we're not concerned about blowing your right foot off either. We've asked for some additional docs. What would be the

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Fujii Masao wrote: On Fri, May 15, 2009 at 8:56 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: When only the history file for timeline 6 is deleted, timeline 6 would be assigned as the newest one *again* at the end of archive recovery. Is this safe? If

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 17:19 +0300, Heikki Linnakangas wrote: Yes, just as deleting old WAL files. So what you're saying is because it's possible to blow your left foot off, we're not concerned about blowing your right foot off either. I don't get it. What are the left

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 17:39 +0300, Heikki Linnakangas wrote: We've asked for some additional docs. What would be the objection to that? I'm certainly not opposed to improving docs. OK, so will you update the docs as requested? -- Simon Riggs www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 10:17 -0400, Andrew Dunstan wrote: This whole area is unfortunately way too fragile. We need some way of managing these facilities that hides a lot of these details and is therefore less likely to produce shot feet, IMNSHO. I get very nervous every time I have to

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Andrew Dunstan
Simon Riggs wrote: On Fri, 2009-05-15 at 10:17 -0400, Andrew Dunstan wrote: This whole area is unfortunately way too fragile. We need some way of managing these facilities that hides a lot of these details and is therefore less likely to produce shot feet, IMNSHO. I get very nervous

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 17:39 +0300, Heikki Linnakangas wrote: We've asked for some additional docs. What would be the objection to that? I'm certainly not opposed to improving docs. OK, so will you update the docs as requested? Well, we already have this in the docs:

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 11:19 -0400, Andrew Dunstan wrote: I don't mean that it has bugs. I mean that it's far too easy to get it wrong and far too hard to get it right. I have reduced my uses to a couple of cases where I have worked out, with some trial and error, recipes that I follow. If

Re: [HACKERS] Testing of parallel restore with current snapshot

2009-05-15 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: Andrew's latest algorithm tends to result in building indexes on the same table at the same time. This is excellent for most users; I'm on a client's site which is I/O bound and that approach is speeding up parallel load about 20% compared to the beta1

Re: [HACKERS] Testing of parallel restore with current snapshot

2009-05-15 Thread Andrew Dunstan
Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Andrew's latest algorithm tends to result in building indexes on the same table at the same time. This is excellent for most users; I'm on a client's site which is I/O bound and that approach is speeding up parallel load about 20%

Re: [HACKERS] Testing of parallel restore with current snapshot

2009-05-15 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: I don't want to mess with it right now either, but perhaps we should have a TODO item to improve the intelligence of parallel restore so that it really does try to do things this way. Other things being equal it schedules things in

Re: [HACKERS] Testing of parallel restore with current snapshot

2009-05-15 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: I don't want to mess with it right now either, but perhaps we should have a TODO item to improve the intelligence of parallel restore so that it really does try to do things this way. Other things

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 18:46 +0300, Heikki Linnakangas wrote: Well, we already have this in the docs: Each time a new timeline is created, PostgreSQL creates a timeline history file that shows which timeline it branched off from and when. These history files are necessary to allow the

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 18:46 +0300, Heikki Linnakangas wrote: What exactly do you want to change? Patch, please. I find this exchange between us quite strange. The discussion on this thread has been fairly clear. Fujii-san and myself have both asked for it to be documented

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Simon Riggs wrote: Do I really need to write a patch to say that, have you formally review it, then change the wording to what you would have written in the first place and then commit? Really? Yes. It's not a trivial change for

Re: [HACKERS] Testing of parallel restore with current snapshot

2009-05-15 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Tom Lane wrote: Based on this thought, what seems to make sense as a quick-and-dirty answer is to make sure that items get scheduled in the same order they came free from dependency restrictions. I don't recall whether that is true at the moment, or

Re: [HACKERS] Testing of parallel restore with current snapshot

2009-05-15 Thread Josh Berkus
Tom, Is anyone interested enough to try it if I code it? If you're patient for results, sure. I seem to be doing a customer migration or upgrade every week now, so it wouldn't take me long to have a test subject with a fairly complex database. -- Josh Berkus PostgreSQL Experts Inc.

[HACKERS] some more plural messages

2009-05-15 Thread Euler Taveira de Oliveira
Hi, While translating some pg_dump messages I noticed that some messages could be part of plural form. I attached a patch that catches those remaining messages. -- Euler Taveira de Oliveira http://www.timbira.com/ Index: src/bin/pg_dump/pg_dump.c