Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-16 Thread Simon Riggs
On Fri, 2009-05-15 at 18:03 -0400, Tom Lane wrote: > I didn't read this thread earlier, but now that I have, it seems to be > making a mountain out of a molehill. We've discussed a complex issue to pursue other nascent bugs. It's confused all of us at some point, but seems we're thru that now.

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Tom Lane
Heikki Linnakangas writes: > Simon Riggs wrote: >> Do I really need to write a patch to say that, have you formally review >> it, then change the wording to what you would have written in the first >> place and then commit? Really? > Yes. It's not a trivial change for me, you're much better at wr

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 18:46 +0300, Heikki Linnakangas wrote: What exactly do you want to change? Patch, please. I find this exchange between us quite strange. The discussion on this thread has been fairly clear. Fujii-san and myself have both asked for it to be documented th

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 18:46 +0300, Heikki Linnakangas wrote: > Well, we already have this in the docs: > > > Each time a new timeline is created, PostgreSQL creates a "timeline > history" file that shows which timeline it branched off from and when. > These history files are necessary to allow t

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 11:19 -0400, Andrew Dunstan wrote: > I don't mean that it has bugs. I mean that it's far too easy to get it > wrong and far too hard to get it right. I have reduced my uses to a > couple of cases where I have worked out, with some trial and error, > recipes that I follow.

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 17:39 +0300, Heikki Linnakangas wrote: We've asked for some additional docs. What would be the objection to that? I'm certainly not opposed to improving docs. OK, so will you update the docs as requested? Well, we already have this in the docs: Ea

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Andrew Dunstan
Simon Riggs wrote: On Fri, 2009-05-15 at 10:17 -0400, Andrew Dunstan wrote: This whole area is unfortunately way too fragile. We need some way of managing these facilities that hides a lot of these details and is therefore less likely to produce shot feet, IMNSHO. I get very nervous ever

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 10:17 -0400, Andrew Dunstan wrote: > This whole area is unfortunately way too fragile. We need some way of > managing these facilities that hides a lot of these details and is > therefore less likely to produce shot feet, IMNSHO. I get very nervous > every time I have to t

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 17:39 +0300, Heikki Linnakangas wrote: > > We've asked for some additional docs. What would be the objection to > > that? > > I'm certainly not opposed to improving docs. OK, so will you update the docs as requested? -- Simon Riggs www.2ndQuadrant.com Postgre

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 17:19 +0300, Heikki Linnakangas wrote: Yes, just as deleting old WAL files. So what you're saying is because it's possible to blow your left foot off, we're not concerned about blowing your right foot off either. I don't get it. What are the left and

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Fujii Masao wrote: On Fri, May 15, 2009 at 8:56 PM, Heikki Linnakangas wrote: Fujii Masao wrote: When only the history file for timeline 6 is deleted, timeline 6 would be assigned as the newest one *again* at the end of archive recovery. Is this safe? If you delete history file and all the WA

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 17:19 +0300, Heikki Linnakangas wrote: > Yes, just as deleting old WAL files. So what you're saying is because it's possible to blow your left foot off, we're not concerned about blowing your right foot off either. We've asked for some additional docs. What would be the ob

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 22:56 +0900, Fujii Masao wrote: OK, I probably understood your point. The timeline history files whose timeline ID is larger than that of an oldest backup must not be deleted from the archive. On the other hand, the smaller or equal one can be deleted. N

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Andrew Dunstan
Simon Riggs wrote: On Fri, 2009-05-15 at 22:56 +0900, Fujii Masao wrote: OK, I probably understood your point. The timeline history files whose timeline ID is larger than that of an oldest backup must not be deleted from the archive. On the other hand, the smaller or equal one can be delet

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 22:56 +0900, Fujii Masao wrote: > OK, I probably understood your point. The timeline history files whose > timeline ID is larger than that of an oldest backup must not be deleted > from the archive. On the other hand, the smaller or equal one can be > deleted. Not all histor

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 15, 2009 at 8:56 PM, Heikki Linnakangas wrote: > Fujii Masao wrote: >> >> On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas >> wrote: >>> >>> The probe in findNewestTimeLine() initialized to recovery target timeline >>> + >>> 1. It doesn't require history files for any old time

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 15:34 +0200, Mikael Krantz wrote: > On Fri, May 15, 2009 at 2:26 PM, Heikki Linnakangas > wrote: > > That was the original issue you ran into. That has now been fixed by forcing > > an xlog switch at pg_start_backup(), so that you can't start a backup in a > > WAL file that

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Mikael Krantz
On Fri, May 15, 2009 at 2:26 PM, Heikki Linnakangas wrote: > That was the original issue you ran into. That has now been fixed by forcing > an xlog switch at pg_start_backup(), so that you can't start a backup in a > WAL file that contains records from a lower numbered timeline. Ah, sorry. /M -

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 15:41 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Fri, 2009-05-15 at 12:56 +0100, Simon Riggs wrote: > > > >> There is no particular reason to send history files to the archive, > >> since new ones are only ever generated at the end of an archive > >> recove

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 15, 2009 at 9:22 PM, Simon Riggs wrote: >> If you delete history file and all the WAL for timeline 6, yeah, nothing >> stops it from being reused. It will work just fine, as if it never >> existed. If you still have the history file and WAL for the old timeline >> 6 lying around s

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 12:56 +0100, Simon Riggs wrote: There is no particular reason to send history files to the archive, since new ones are only ever generated at the end of an archive recovery. It also clears up a long standing confusion between backup history files and t

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: ehem, "It will work fine" isn't correct, as Fujii-san observes. What exactly are the steps required to run into that problem? I fail to see what the problem is. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pg

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Mikael Krantz
On Fri, May 15, 2009 at 2:22 PM, Simon Riggs wrote: > Let's document that timeline files should not be deleted from the > archive iff there exists a base backup made during a lower numbered > timeline. Or made during a higher numbered timeline which happens to start in a WAL-file containing recor

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Mikael Krantz wrote: On Fri, May 15, 2009 at 2:22 PM, Simon Riggs wrote: Let's document that timeline files should not be deleted from the archive iff there exists a base backup made during a lower numbered timeline. Or made during a higher numbered timeline which happens to start in a WAL-fi

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 14:56 +0300, Heikki Linnakangas wrote: > Simon's idea of keeping a copy of all the history files in the data > directory wouldn't help here. In fact, I think we already never delete > history files in the server, it's just that if you omit the pg_xlog > directory in the b

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Simon Riggs wrote: On Fri, 2009-05-15 at 20:38 +0900, Fujii Masao wrote: On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas wrote: The probe in findNewestTimeLine() initialized to recovery target timeline + 1. It doesn't require history files for any old timelines to be present. What if rec

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Fujii Masao wrote: On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas wrote: The probe in findNewestTimeLine() initialized to recovery target timeline + 1. It doesn't require history files for any old timelines to be present. What if recovery_target_timeline = 'latest'? The unexpected (not l

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 12:56 +0100, Simon Riggs wrote: > There is no particular reason to send history files to the archive, > since new ones are only ever generated at the end of an archive > recovery. It also clears up a long standing confusion between backup history files and timeline history

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 20:38 +0900, Fujii Masao wrote: > On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas > wrote: > > The probe in findNewestTimeLine() initialized to recovery target timeline + > > 1. It doesn't require history files for any old timelines to be present. > > What if recovery_

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 15, 2009 at 8:20 PM, Heikki Linnakangas wrote: > The probe in findNewestTimeLine() initialized to recovery target timeline + > 1. It doesn't require history files for any old timelines to be present. What if recovery_target_timeline = 'latest'? The unexpected (not latest) recover

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Simon Riggs
On Fri, 2009-05-15 at 20:11 +0900, Fujii Masao wrote: > Hi, > > On Fri, May 8, 2009 at 2:42 AM, Heikki Linnakangas > wrote: > > When you create a new base backup, you shouldn't need any files archived > > before starting the backup. > > If so, this fix is not enough, since findNewestTimeLine()

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Heikki Linnakangas
Fujii Masao wrote: Hi, On Fri, May 8, 2009 at 2:42 AM, Heikki Linnakangas wrote: When you create a new base backup, you shouldn't need any files archived before starting the backup. If so, this fix is not enough, since findNewestTimeLine() is still based on the premise that *all* the history

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-15 Thread Fujii Masao
Hi, On Fri, May 8, 2009 at 2:42 AM, Heikki Linnakangas wrote: > When you create a new base backup, you shouldn't need any files archived > before starting the backup. If so, this fix is not enough, since findNewestTimeLine() is still based on the premise that *all* the history files exist. So, a

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-07 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-07 at 18:57 +0300, Heikki Linnakangas wrote: I don't see any user error here. Just observing that the error occurs because we rely on a file being there when we haven't even documented that it needs to be there for it to work. File deletion with %r from the a

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 18:57 +0300, Heikki Linnakangas wrote: > I don't see any user error here. Just observing that the error occurs because we rely on a file being there when we haven't even documented that it needs to be there for it to work. File deletion with %r from the archive would not hav

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-07 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-07 at 17:54 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: A more useful thing might be to do an xlog switch before we do the shutdown checkpoint at end of recovery. That gives the same sequence of actions without modifying the existing sequence of activi

Re: [HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 17:54 +0300, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Thu, 2009-05-07 at 12:15 +0300, Heikki Linnakangas wrote: > > > >> Yeah, I think you're right. If you omit pg_xlog from the base backup, > >> as we recommend in the manual, and clear the old files from the >

[HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-07 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-07 at 12:15 +0300, Heikki Linnakangas wrote: Yeah, I think you're right. If you omit pg_xlog from the base backup, as we recommend in the manual, and clear the old files from the archive too, then you won't have the old history file around. ... A more useful

[HACKERS] Re: [BUGS] BUG #4796: Recovery followed by backup creates unrecoverable WAL-file

2009-05-07 Thread Simon Riggs
On Thu, 2009-05-07 at 12:15 +0300, Heikki Linnakangas wrote: > Yeah, I think you're right. If you omit pg_xlog from the base backup, > as we recommend in the manual, and clear the old files from the > archive too, then you won't have the old history file around. Sorry about this, but I don't agr