Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2009-01-27 Thread Heikki Linnakangas
(replying to a very old message, since I just bumped into this in review) Alvaro Herrera wrote: Simon Riggs wrote: On Fri, 2008-09-12 at 14:14 -0400, Alvaro Herrera wrote: Simon Riggs wrote: --- 5716,5725 CheckpointStats.ckpt_sync_end_t,

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2009-01-27 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: (replying to a very old message, since I just bumped into this in review) Alvaro Herrera wrote: The new message is not translatable, the original was. Doesn't really matter since it's an elog(), not ereport(). ... which is wrong

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Thu, 2008-10-02 at 19:07 -0400, Andrew Dunstan wrote: Simon Riggs wrote: Just seen this patch has been bounced into November CommitFest, even though the new patch fixes all of the concerns raised. I'm concerned that this is going to make the final Hot Standby patch fairly large,

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: A patch specifically marked as required for other work has been delayed by more than 5 weeks on queue and nobody was ever assigned to review it. That was exactly the problem CommitFests were supposed to resolve and from my perspective this is a systemic

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 10:08 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: A patch specifically marked as required for other work has been delayed by more than 5 weeks on queue and nobody was ever assigned to review it. That was exactly the problem CommitFests were supposed

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Alvaro Herrera
Simon Riggs wrote: I'm just grumpy because I can't see a way to do the patch-on-patch-on-patch that I'll need to make this all work for Nov 1. So big patch here we come. But that's just the way it is and I'll stop honking about it. This is one of the problems that DVCSs are supposed to solve

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Robert Haas
I'm just grumpy because I can't see a way to do the patch-on-patch-on-patch that I'll need to make this all work for Nov 1. So big patch here we come. But that's just the way it is and I'll stop honking about it. This is one of the problems that DVCSs are supposed to solve ... have you

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 10:37 -0400, Alvaro Herrera wrote: Simon Riggs wrote: I'm just grumpy because I can't see a way to do the patch-on-patch-on-patch that I'll need to make this all work for Nov 1. So big patch here we come. But that's just the way it is and I'll stop honking about

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Tom Lane
Robert Haas [EMAIL PROTECTED] writes: I'm just grumpy because I can't see a way to do the patch-on-patch-on-patch that I'll need to make this all work for Nov 1. So big patch here we come. But that's just the way it is and I'll stop honking about it. This is one of the problems that DVCSs

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Dave Page
On Tue, Oct 7, 2008 at 4:08 PM, Simon Riggs [EMAIL PROTECTED] wrote: Not convinced this is the right time to invest in side activities, but if you think so, I'll look into it. Anybody wanting to write or link to a Simon's Guide, most welcome. Heikki will be presenting a talk about GIT + PG

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Heikki Linnakangas
Alvaro Herrera wrote: Simon Riggs wrote: I'm just grumpy because I can't see a way to do the patch-on-patch-on-patch that I'll need to make this all work for Nov 1. So big patch here we come. But that's just the way it is and I'll stop honking about it. This is one of the problems that DVCSs

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-07 Thread Simon Riggs
On Tue, 2008-10-07 at 18:19 +0300, Heikki Linnakangas wrote: The bottom line is that hot standby is a big feature, and probably a big patch. No amount of version control will work around that. Finishing all that in a few weeks is a very ambitious goal. I wish you luck, and I wish I could

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-02 Thread Simon Riggs
On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Version 7 After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into consistent recovery mode as soon as it's read WAL beyond

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-10-02 Thread Andrew Dunstan
Simon Riggs wrote: On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Version 7 After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into consistent recovery mode as soon as it's read

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Sun, 2008-09-28 at 21:16 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: It does nothing AFAICS for the problem that when restarting archive recovery from a restartpoint, it's not clear when it is safe to start letting in backends. You need to get past the highest LSN

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: ... That kinda works, but the problem is that restartpoints are time based, not log based. We need them to be deterministic for us to rely upon them in the above way. Right, but the performance disadvantages of making them strictly log-distance-based are

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 08:46 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: ... That kinda works, but the problem is that restartpoints are time based, not log based. We need them to be deterministic for us to rely upon them in the above way. Right, but the performance

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: I think we can get away with writing the LSN value to disk, as you suggested, but only every so often. No need to do it after every WAL record, just consistently every so often, so it gives us a point at which we know we are safe. Huh? How does that make

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: I think we can get away with writing the LSN value to disk, as you suggested, but only every so often. No need to do it after every WAL record, just consistently every so often, so it gives us a point

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: ... If we crash and restart, we'll have to get to the end of this file before we start letting backends in; which might be further than we actually got before the crash, but not too much further because

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-29 Thread Simon Riggs
On Mon, 2008-09-29 at 11:24 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Mon, 2008-09-29 at 10:13 -0400, Tom Lane wrote: ... If we crash and restart, we'll have to get to the end of this file before we start letting backends in; which might be further than we actually

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-28 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into consistent recovery mode as soon as it's read WAL beyond ControlFile-minRecoveryPoint. In a

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-28 Thread Simon Riggs
On Sun, 2008-09-28 at 14:02 -0400, Tom Lane wrote: It does nothing AFAICS for the problem that when restarting archive recovery from a restartpoint, it's not clear when it is safe to start letting in backends. You need to get past the highest LSN that has made it out to disk, and there is

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-28 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: It does nothing AFAICS for the problem that when restarting archive recovery from a restartpoint, it's not clear when it is safe to start letting in backends. You need to get past the highest LSN that has made it out to disk, and there is no good way to

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-26 Thread Simon Riggs
On Thu, 2008-09-25 at 18:28 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Version 7 Anyway, that's sufficiently bad that I'm bouncing the patch for reconsideration. No problem, I understand this needs discussion. There's less detail here than first appears. There are

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-26 Thread Simon Riggs
On Fri, 2008-09-26 at 11:20 +0100, Simon Riggs wrote: After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into consistent recovery mode as soon as it's read WAL beyond ControlFile-minRecoveryPoint. In a crash recovery situation that

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-25 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: Version 7 After reading this for awhile, I realized that there is a rather fundamental problem with it: it switches into consistent recovery mode as soon as it's read WAL beyond ControlFile-minRecoveryPoint. In a crash recovery situation that typically is

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-23 Thread Simon Riggs
On Mon, 2008-09-22 at 23:06 +0100, Simon Riggs wrote: On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: Do we really need a checkpoint there at all? Timelines only change at shutdown

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-22 Thread Simon Riggs
On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: Do we really need a checkpoint there at all? Timelines only change at shutdown checkpoints. Hmm. I *think* that that is just a debugging

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Tue, 2008-09-16 at 15:45 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] wrote: Testing takes a while on this, I probably won't complete it until Friday. So enclosed patch is for eyeballs only at this stage. What's the status on that patch? Having some trouble trying to get a

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 09:05 +0100, Simon Riggs wrote: Feels like I should shutdown the bgwriter after recovery and then allow it to be cranked up again after normal processing starts, and do all of this through postmaster state changes. That way bgwriter doesn't need to do a dynamic state

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: Having some trouble trying to get a clean state change from recovery to normal mode. Startup needs to be able to write WAL at the end of recovery so it can write a ShutdownCheckpoint, yet must not be allowed to write WAL before that. Other services are

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Having some trouble trying to get a clean state change from recovery to normal mode. Startup needs to be able to write WAL at the end of recovery so it can write a ShutdownCheckpoint, yet must not

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: Do we really need a checkpoint there at all? Timelines only change at shutdown checkpoints. Hmm. I *think* that that is just a debugging crosscheck rather than a critical property. But yeah, it would

Re: [PATCHES] [HACKERS] Infrastructure changes for recovery

2008-09-18 Thread Simon Riggs
On Thu, 2008-09-18 at 10:09 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: On Thu, 2008-09-18 at 09:06 -0400, Tom Lane wrote: Do we really need a checkpoint there at all? Timelines only change at shutdown checkpoints. Hmm. I *think* that that is just a debugging