Re: [HACKERS] Clean shutdown and warm standby

2009-05-30 Thread Simon Riggs
On Fri, 2009-05-29 at 21:16 -0300, Euler Taveira de Oliveira wrote: Simon Riggs escreveu: And for them, it hasn't been completely fixed. That point was not made by patch author or committer, leaving the impression it was now completely safe, which, I truly regret to say, is not correct.

Re: [HACKERS] Clean shutdown and warm standby

2009-05-30 Thread Euler Taveira de Oliveira
Simon Riggs escreveu: And for them, it hasn't been completely fixed. That point was not made by patch author or committer, leaving the impression it was now completely safe, which, I truly regret to say, is not correct. Simon, could you point out what the patch does not do? If we can't fix it

Re: [HACKERS] Clean shutdown and warm standby

2009-05-29 Thread Simon Riggs
On Thu, 2009-05-28 at 18:09 -0400, Tom Lane wrote: What's your point? Surely the applied patch is a *necessary* component of any attempt to try to ensure archiving is complete at shutdown. I agree that it doesn't cover every risk factor, and there are some risk factors that cannot be covered

Re: [HACKERS] Clean shutdown and warm standby

2009-05-29 Thread Heikki Linnakangas
Simon Riggs wrote: Regrettably, the patch doesn't remove the problem it was supposed to remove and I'm highlighting there is still risk of data loss. I feel that you're moving the goalposts. What exactly is the problem it was supposed to remove in your opinion? -- Heikki Linnakangas

Re: [HACKERS] Clean shutdown and warm standby

2009-05-29 Thread Robert Haas
On Fri, May 29, 2009 at 2:23 PM, Simon Riggs si...@2ndquadrant.com wrote: Regrettably, the patch doesn't remove the problem it was supposed to remove and I'm highlighting there is still risk of data loss. I suggest that we don't change any docs, and carefully word or even avoid any release

Re: [HACKERS] Clean shutdown and warm standby

2009-05-29 Thread Simon Riggs
On Fri, 2009-05-29 at 21:46 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: Regrettably, the patch doesn't remove the problem it was supposed to remove and I'm highlighting there is still risk of data loss. I feel that you're moving the goalposts. What exactly is the problem it was

Re: [HACKERS] Clean shutdown and warm standby

2009-05-29 Thread Simon Riggs
On Fri, 2009-05-29 at 14:54 -0400, Robert Haas wrote: On Fri, May 29, 2009 at 2:23 PM, Simon Riggs si...@2ndquadrant.com wrote: Regrettably, the patch doesn't remove the problem it was supposed to remove and I'm highlighting there is still risk of data loss. I suggest that we don't change

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Heikki Linnakangas
Guillaume Smet wrote: On Tue, Apr 28, 2009 at 5:35 PM, Guillaume Smet guillaume.s...@gmail.com wrote: On Tue, Apr 28, 2009 at 5:22 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: At a normal startup, the checkpoint record would be there as usual. And an archive recovery

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 14:04 +0300, Heikki Linnakangas wrote: I've committed a patch to do the RequstXLogSwitch() before shutdown checkpoint as discussed. It seems safe to me. (sorry for the delay, and thanks for the reminder) Not sure if that is a fix that will work in all cases. There

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-28 at 14:04 +0300, Heikki Linnakangas wrote: I've committed a patch to do the RequstXLogSwitch() before shutdown checkpoint as discussed. It seems safe to me. (sorry for the delay, and thanks for the reminder) Not sure if that is a fix that will work in

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 16:19 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-05-28 at 14:04 +0300, Heikki Linnakangas wrote: I've committed a patch to do the RequstXLogSwitch() before shutdown checkpoint as discussed. It seems safe to me. (sorry for the delay, and

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-28 at 16:19 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-05-28 at 14:04 +0300, Heikki Linnakangas wrote: I've committed a patch to do the RequstXLogSwitch() before shutdown checkpoint as discussed. It seems safe to me. (sorry for the

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 16:52 +0300, Heikki Linnakangas wrote: If the archiver is working, but has fallen behind at the point of shutdown, does the archiver operate for long enough to ensure we are archived up to the point of the log switch prior to checkpoint? Yes, it archives all

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-28 at 16:52 +0300, Heikki Linnakangas wrote: If the archiver is working, but has fallen behind at the point of shutdown, does the archiver operate for long enough to ensure we are archived up to the point of the log switch prior to checkpoint? Yes, it

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 17:21 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I don't think it does, please look again. Still looks ok to me. pgarch_ArchiverCopyLoop() loops until all ready WAL segments have been archived (assuming no errors). No, it doesn't now, though it did used

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-28 at 17:21 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I don't think it does, please look again. Still looks ok to me. pgarch_ArchiverCopyLoop() loops until all ready WAL segments have been archived (assuming no errors). No, it doesn't now, though

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Andreas Pflug
Simon Riggs wrote: No, because as I said, if archive_command has been returning non-zero then the archive will be incomplete. Yes. You think that's wrong? How would you like it to behave, then? I don't think you want the shutdown to wait indefinitely until all files have been

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Guillaume Smet
On Thu, May 28, 2009 at 5:02 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So you check. This solves Guillaume's immediate concern. If you have a suggestion for further improvements, I'm all ears. Thanks for applying the patch. Yes, the problem is that before this change,

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 17:16 +0200, Guillaume Smet wrote: On Thu, May 28, 2009 at 5:02 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So you check. This solves Guillaume's immediate concern. If you have a suggestion for further improvements, I'm all ears. Thanks for

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Guillaume Smet
On Thu, May 28, 2009 at 5:36 PM, Simon Riggs si...@2ndquadrant.com wrote: If you feel we have moved forwards, that's good, but since no part of the *safe* maintenance procedure has changed, I don't see that myself. Only the unsafe way of doing it got faster. I disagree with you. The situation

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 17:50 +0200, Guillaume Smet wrote: I think it's a step forward, maybe not sufficient for you but I prefer the situation now than before. It's safer because of the principle of least surprise: I'm pretty sure a lot of people didn't even think that the last WAL file was

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Guillaume Smet
On Thu, May 28, 2009 at 6:06 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2009-05-28 at 17:50 +0200, Guillaume Smet wrote: I think it's a step forward, maybe not sufficient for you but I prefer the situation now than before. It's safer because of the principle of least surprise: I'm

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 18:02 +0300, Heikki Linnakangas wrote: postmaster never sends SIGTERM to pgarch, and postmaster is still alive. Then we have a regression, since we changed the code to make sure the archiver did shutdown even if there was a backlog. The reason is that if there is a long

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-05-28 at 18:02 +0300, Heikki Linnakangas wrote: postmaster never sends SIGTERM to pgarch, and postmaster is still alive. Then we have a regression, since we changed the code to make sure the archiver did shutdown even if there was a backlog. The commit

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Simon Riggs
On Thu, 2009-05-28 at 19:58 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-05-28 at 18:02 +0300, Heikki Linnakangas wrote: postmaster never sends SIGTERM to pgarch, and postmaster is still alive. Then we have a regression, since we changed the code to make sure the

Re: [HACKERS] Clean shutdown and warm standby

2009-05-28 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Thus there is no guarantee that this is sufficient to have archived all the files you would like to archive. The patch does not provide a clean shutdown in all cases and since you don't know what state its in, you are still forced to take external

Re: [HACKERS] Clean shutdown and warm standby

2009-05-27 Thread Guillaume Smet
On Tue, Apr 28, 2009 at 5:35 PM, Guillaume Smet guillaume.s...@gmail.com wrote: On Tue, Apr 28, 2009 at 5:22 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: At a normal startup, the checkpoint record would be there as usual. And an archive recovery starts at the location

Re: [HACKERS] Clean shutdown and warm standby

2009-04-28 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Andreas Pflug wrote: So to get this down to a solution, it appears to be correct to execute the RequestXLogSwitch right before CreateCheckPoint? Hmm, then the checkpoint record isn't archived. That might be acceptable, though,

Re: [HACKERS] Clean shutdown and warm standby

2009-04-28 Thread Andreas Pflug
Tom Lane wrote: Not at all, because the database would be very unhappy at restart if it can't find the checkpoint record pg_control is pointing to. So for several weeks now all postings just say how it will _not_ work. Does this boil down to There's no way to make sure that a graceful

Re: [HACKERS] Clean shutdown and warm standby

2009-04-28 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Andreas Pflug wrote: So to get this down to a solution, it appears to be correct to execute the RequestXLogSwitch right before CreateCheckPoint? Hmm, then the checkpoint record isn't archived. That might be

Re: [HACKERS] Clean shutdown and warm standby

2009-04-28 Thread Guillaume Smet
On Tue, Apr 28, 2009 at 5:22 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: At a normal startup, the checkpoint record would be there as usual. And an archive recovery starts at the location indicated by the backup label. AFAICS calling RequestXLogSwitch() before

Re: [HACKERS] Clean shutdown and warm standby

2009-04-27 Thread Heikki Linnakangas
Fujii Masao wrote: On Fri, Apr 24, 2009 at 3:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It's not safe to write WAL after the checkpoint, as RequestXLogSwitch() does. After restart, the system will start inserting WAL from the checkpoint redo point, which is just before

Re: [HACKERS] Clean shutdown and warm standby

2009-04-27 Thread Fujii Masao
Hi, On Fri, Apr 24, 2009 at 3:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It's not safe to write WAL after the checkpoint, as RequestXLogSwitch() does. After restart, the system will start inserting WAL from the checkpoint redo point, which is just before the

Re: [HACKERS] Clean shutdown and warm standby

2009-04-27 Thread Fujii Masao
Hi, On Mon, Apr 27, 2009 at 8:43 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: On Fri, Apr 24, 2009 at 3:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It's not safe to write WAL after the checkpoint, as RequestXLogSwitch() does.

Re: [HACKERS] Clean shutdown and warm standby

2009-04-27 Thread Heikki Linnakangas
Fujii Masao wrote: Hi, On Mon, Apr 27, 2009 at 8:43 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: On Fri, Apr 24, 2009 at 3:20 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It's not safe to write WAL after the checkpoint, as

Re: [HACKERS] Clean shutdown and warm standby

2009-04-27 Thread Fujii Masao
Hi, On Mon, Apr 27, 2009 at 10:02 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: Hi, On Mon, Apr 27, 2009 at 8:43 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: On Fri, Apr 24, 2009 at 3:20 PM, Heikki

Re: [HACKERS] Clean shutdown and warm standby

2009-04-27 Thread Andreas Pflug
Heikki Linnakangas wrote: No, no crash is involved. Just a normal server shutdown and start: 1. Server shutdown is initiated 2. A shutdown checkpoint is recorded at XLOG point 1234, redo ptr is also 1234. 3. A XLOG_SWITCH record is written at 1235, right after the checkpoint record. 4. The

Re: [HACKERS] Clean shutdown and warm standby

2009-04-27 Thread Heikki Linnakangas
Andreas Pflug wrote: Heikki Linnakangas wrote: No, no crash is involved. Just a normal server shutdown and start: 1. Server shutdown is initiated 2. A shutdown checkpoint is recorded at XLOG point 1234, redo ptr is also 1234. 3. A XLOG_SWITCH record is written at 1235, right after the

Re: [HACKERS] Clean shutdown and warm standby

2009-04-24 Thread Heikki Linnakangas
Guillaume Smet wrote: On Wed, Apr 8, 2009 at 9:11 PM, I wrote: Following the discussion here http://archives.postgresql.org/message-id/49d9e986.8010...@pse-consulting.de , I wrote a small patch which rotates the last XLog file on shutdown [snip] Any comment or advice on how I can fix it with

Re: [HACKERS] Clean shutdown and warm standby

2009-04-14 Thread Guillaume Smet
Hi, On Wed, Apr 8, 2009 at 9:11 PM, I wrote: Following the discussion here http://archives.postgresql.org/message-id/49d9e986.8010...@pse-consulting.de , I wrote a small patch which rotates the last XLog file on shutdown [snip] Any comment or advice on how I can fix it with a different

Re: [HACKERS] Clean shutdown and warm standby

2009-04-09 Thread Guillaume Smet
On Thu, Apr 9, 2009 at 5:00 AM, Fujii Masao masao.fu...@gmail.com wrote: RequestXLogSwitch() doesn't wait until the switched WAL file has actually been archived. So, some WAL files still may not exist in the standby server also after clean shutdown of the primary. Thanks for your comment.

Re: [HACKERS] Clean shutdown and warm standby

2009-04-09 Thread Fujii Masao
Hi, On Thu, Apr 9, 2009 at 6:36 PM, Guillaume Smet guillaume.s...@gmail.com wrote: On Thu, Apr 9, 2009 at 5:00 AM, Fujii Masao masao.fu...@gmail.com wrote: RequestXLogSwitch() doesn't wait until the switched WAL file has actually been archived. So, some WAL files still may not exist in the

Re: [HACKERS] Clean shutdown and warm standby

2009-04-08 Thread Fujii Masao
Hi, On Thu, Apr 9, 2009 at 4:11 AM, Guillaume Smet guillaume.s...@gmail.com wrote: Hi, Following the discussion here http://archives.postgresql.org/message-id/49d9e986.8010...@pse-consulting.de , I wrote a small patch which rotates the last XLog file on shutdown so that the archive command