Re: [HACKERS] Hot standby?

2009-08-11 Thread Josh Berkus
Peter, I believe we're just copying Oracle's terminology. While that terminology is not consistent, it is understood by the industry. Oracle defined their Hot Standby to have both asynchronous and synchronous modes: http://www.oracle.com/technology/products/rdb/htdocs/dbms/hotstandby.html The

Re: [HACKERS] Hot standby?

2009-08-11 Thread Robert Haas
On Tue, Aug 11, 2009 at 1:08 PM, Josh Berkusj...@agliodbs.com wrote: I believe we're just copying Oracle's terminology.  While that terminology is not consistent, it is understood by the industry.  Oracle defined their Hot Standby to have both asynchronous and synchronous modes:

Re: [HACKERS] Hot standby?

2009-08-11 Thread Ron Mayer
David Fetter wrote: On Tue, Aug 11, 2009 at 08:56:38AM -0500, Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: OK, so it is warm slave. Why isn't it just a read only slave. Do some systems have read-only slave databases that can't serve as a warm standby system as well as this

Re: [HACKERS] Hot standby?

2009-08-11 Thread Mark Mielke
On 08/11/2009 09:56 AM, Kevin Grittner wrote: Bruce Momjianbr...@momjian.us wrote: OK, so it is warm slave. That is technically accurate, given the preceding definitions, but it has disturbing connotations. Enough so, in my view, to merit getting a little more creative in the

Re: [HACKERS] Hot standby?

2009-08-11 Thread Robert Haas
On Tue, Aug 11, 2009 at 2:48 PM, Mark Mielkem...@mark.mielke.cc wrote: On 08/11/2009 09:56 AM, Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: OK, so it is warm slave. That is technically accurate, given the preceding definitions, but it has disturbing connotations. Enough

Re: [HACKERS] Hot standby?

2009-08-11 Thread Robert Haas
On Tue, Aug 11, 2009 at 2:10 PM, Ron Mayerrm...@cheapcomplexdevices.com wrote: David Fetter wrote: On Tue, Aug 11, 2009 at 08:56:38AM -0500, Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: OK, so it is warm slave. Why isn't it just a read only slave.  Do some systems have

Re: [HACKERS] Hot standby?

2009-08-11 Thread Mark Mielke
On 08/11/2009 02:52 PM, Robert Haas wrote: On Tue, Aug 11, 2009 at 2:48 PM, Mark Mielkem...@mark.mielke.cc wrote: I remember this debate from 6 months ago. :-) I prefer not to try and fit square pegs into round holes. Streaming replication sounds like the best description. It may not be

Re: [HACKERS] Hot standby?

2009-08-11 Thread Alvaro Herrera
Mark Mielke escribió: I don't think I was confused before - but I am confused now. :-) This patch does not provide streaming replication? No. What it does is allow you to query the slave while it's still replaying transactions. There's another patch allowing you to do stream replication.

Re: [HACKERS] Hot standby?

2009-08-11 Thread Kevin Grittner
Mark Mielke m...@mark.mielke.cc wrote: This patch does not provide streaming replication? There's a separate effort to provide asynchronous and synchronous streaming replication. Different patch. Hot standby to me means the slave is as close to up-to-date as possible and can potentially

Re: [HACKERS] Hot standby?

2009-08-11 Thread Josh Berkus
All, So really, the streaming replication patch should be called hot standby, and the hot standby patch should be called read only slaves? And *why* can't we call it log-based replication? -- Josh Berkus PostgreSQL Experts Inc. www.pgexperts.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Hot standby?

2009-08-11 Thread Merlin Moncure
On Tue, Aug 11, 2009 at 4:07 PM, Josh Berkusj...@agliodbs.com wrote: All, So really, the streaming replication patch should be called hot standby, and the hot standby patch should be called read only slaves? And *why* can't we call it log-based replication? +1 *) it _is_ used to replicate

Re: [HACKERS] Hot standby?

2009-08-11 Thread Gianni Ciolli
On Tue, Aug 11, 2009 at 01:14:56PM -0400, Robert Haas wrote: On Tue, Aug 11, 2009 at 1:08 PM, Josh Berkusj...@agliodbs.com wrote: I believe we're just copying Oracle's terminology.  While that terminology is not consistent, it is understood by the industry.  Oracle defined their Hot Standby

Re: [HACKERS] Hot standby?

2009-08-11 Thread Peter Eisentraut
On Tuesday 11 August 2009 18:16:04 Gianni Ciolli wrote: As for warm/hot, it depends on what you exactly mean with get ready: (A) If you mean it is possible to connect to the second node, then Simon's patch is hot. Yeah, but by that definiton doing a pg_dump/pg_restore every hour is also

Re: [HACKERS] Hot standby?

2009-08-11 Thread Robert Haas
On Tue, Aug 11, 2009 at 4:07 PM, Josh Berkusj...@agliodbs.com wrote: So really, the streaming replication patch should be called hot standby, No. AIUI, hot standby means that when your primary falls over, the secondary automatically promotes itself and takes over. It requires things like

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-11 Thread Dimitri Fontaine
Hi, Le 11 août 09 à 07:50, Heikki Linnakangas a écrit : 2009/8/11 Robert Haas robertmh...@gmail.com We should probably have a separate discussion about what the least committable unit would be for this patch. I wonder if it might be sufficient to provide a facility for streaming WAL, plus a

Re: [HACKERS] Hot standby?

2009-08-11 Thread Pierre Frédéric Caillau d
Incidentally, we billed pg_dump as hot backup at some point. mysql calls mysqlhotcopy a script that locks and flushes all tables, then makes a copy of the database directory (all queries being locked out while this is in progress, of course). -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-11 Thread Robert Haas
On Tue, Aug 11, 2009 at 5:20 PM, Dimitri Fontainedfonta...@hi-media.com wrote: We should somehow provide a default archive and restore command integrated into the main product, so that it's as easy as turning it 'on' in the configuration for users to have something trustworthy: PostgreSQL will

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-11 Thread Dimitri Fontaine
Le 11 août 09 à 23:30, Robert Haas a écrit : On Tue, Aug 11, 2009 at 5:20 PM, Dimitri Fontainedfonta...@hi-media.com wrote: We should somehow provide a default archive and restore command integrated into the main product, so that it's as easy as turning it 'on' in the configuration for

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-11 Thread Joshua D. Drake
On Tue, 2009-08-11 at 17:30 -0400, Robert Haas wrote: On Tue, Aug 11, 2009 at 5:20 PM, Dimitri Fontainedfonta...@hi-media.com wrote: We should somehow provide a default archive and restore command integrated into the main product, so that it's as easy as turning it 'on' in the

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-11 Thread Robert Haas
On Tue, Aug 11, 2009 at 5:40 PM, Dimitri Fontainedfonta...@hi-media.com wrote: Le 11 août 09 à 23:30, Robert Haas a écrit : On Tue, Aug 11, 2009 at 5:20 PM, Dimitri Fontainedfonta...@hi-media.com wrote: We should somehow provide a default archive and restore command integrated into the

Re: [HACKERS] Hot standby?

2009-08-11 Thread Andrew Dunstan
Pierre Frédéric Caillaud wrote: Incidentally, we billed pg_dump as hot backup at some point. mysql calls mysqlhotcopy a script that locks and flushes all tables, then makes a copy of the database directory (all queries being locked out while this is in progress, of course). Doesn't

Re: [HACKERS] Hot standby?

2009-08-11 Thread Josh Berkus
Incidentally, we billed pg_dump as hot backup at some point. It *is* hot backup as in taken while the database is running and fully accessible. mysql calls mysqlhotcopy a script that locks and flushes all tables, then makes a copy of the database directory (all queries being locked out

Re: [HACKERS] Hot standby?

2009-08-11 Thread Gianni Ciolli
On Wed, Aug 12, 2009 at 12:11:28AM +0300, Peter Eisentraut wrote: On Tuesday 11 August 2009 18:16:04 Gianni Ciolli wrote: As for warm/hot, it depends on what you exactly mean with get ready: (A) If you mean it is possible to connect to the second node, then Simon's patch is hot.

Re: [HACKERS] Hot standby?

2009-08-11 Thread Bruce Momjian
Robert Haas wrote: On Tue, Aug 11, 2009 at 4:07 PM, Josh Berkusj...@agliodbs.com wrote: So really, the streaming replication patch should be called hot standby, No. AIUI, hot standby means that when your primary falls over, the secondary automatically promotes itself and takes over. It

Re: [HACKERS] Hot standby?

2009-08-11 Thread Greg Stark
On Tue, Aug 11, 2009 at 10:13 PM, Robert Haasrobertmh...@gmail.com wrote: On Tue, Aug 11, 2009 at 4:07 PM, Josh Berkusj...@agliodbs.com wrote: So really, the streaming replication patch should be called hot standby, No.  AIUI, hot standby means that when your primary falls over, the

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-11 Thread Fujii Masao
Hi, On Wed, Aug 12, 2009 at 6:50 AM, Robert Haasrobertmh...@gmail.com wrote: I don't really know how you COULD pick a safe default location. Presumably any location that's in the default postgresql.conf file would be under $PGDATA, which kind of defeats the purpose of the whole thing.  In

Re: [HACKERS] Hot standby?

2009-08-11 Thread Greg Stark
As I see it we potentially have the following modes to deal with: Archive file mode asynchronous archive Archive file mode asynchronous standby slave Streaming mode asynchronous standby slave Streaming mode synchronousstandby slave Archive file mode asynchronous

Re: [HACKERS] Hot standby?

2009-08-11 Thread Robert Haas
On Tue, Aug 11, 2009 at 9:44 PM, Greg Starkgsst...@mit.edu wrote: On Tue, Aug 11, 2009 at 10:13 PM, Robert Haasrobertmh...@gmail.com wrote: On Tue, Aug 11, 2009 at 4:07 PM, Josh Berkusj...@agliodbs.com wrote: So really, the streaming replication patch should be called hot standby, No.  AIUI,

Re: [HACKERS] Hot standby?

2009-08-11 Thread Mark Mielke
On 08/11/2009 11:19 PM, Robert Haas wrote: On Tue, Aug 11, 2009 at 9:44 PM, Greg Starkgsst...@mit.edu wrote: No! This is *not* what hot standby means, at least not in the Oracle world. I'm perplexed by this. For example: http://en.wikipedia.org/wiki/Hot_standby Admittedly,

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-10 Thread Joshua D. Drake
On Sun, 2009-08-09 at 22:15 -0400, Robert Haas wrote: On Sun, Aug 9, 2009 at 2:43 PM, Simon Riggssi...@2ndquadrant.com wrote: I've said very clearly that I am working on this and it's fairly laughable to suggest that anybody thought I wasn't. What more should I do to prove something is

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-10 Thread Josh Berkus
All, Can we stop arguing about a patch everyone wants? Simon: you have people offering to help with the patch. Offering to help *right now*. Might I suggest that you establish a GIT branch for Hot Standby so that more people can collaborate? Working on it until you get it perfect offsite

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-10 Thread Joshua D. Drake
On Mon, 2009-08-10 at 10:20 -0700, Josh Berkus wrote: All, Can we stop arguing about a patch everyone wants? Simon: you have people offering to help with the patch. Offering to help *right now*. Might I suggest that you establish a GIT branch for Hot Standby so that more people can

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-10 Thread Simon Riggs
On Mon, 2009-08-10 at 10:20 -0700, Josh Berkus wrote: Simon: you have people offering to help with the patch. Offering to help *right now*. Might I suggest that you establish a GIT branch for Hot Standby so that more people can collaborate? Working on it until you get it perfect offsite

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-10 Thread David Fetter
On Mon, Aug 10, 2009 at 11:15:51PM +0100, Simon Riggs wrote: On Mon, 2009-08-10 at 10:20 -0700, Josh Berkus wrote: Simon: you have people offering to help with the patch. Offering to help *right now*. Might I suggest that you establish a GIT branch for Hot Standby so that more people

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-10 Thread Bruce Momjian
Simon Riggs wrote: On Mon, 2009-08-10 at 10:20 -0700, Josh Berkus wrote: Simon: you have people offering to help with the patch. Offering to help *right now*. Might I suggest that you establish a GIT branch for Hot Standby so that more people can collaborate? Working on it until

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-10 Thread Robert Haas
On Mon, Aug 10, 2009 at 9:51 PM, Bruce Momjianbr...@momjian.us wrote: What is the status of hot standby and synchronous replication?  Is there a design specification?  Who are the lead developers?  Who is assisting? What open item do we have for each feature?  Where is the most recent patch?  

Re: [HACKERS] Hot standby and synchronous replication status

2009-08-10 Thread Heikki Linnakangas
2009/8/11 Robert Haas robertmh...@gmail.com We should probably have a separate discussion about what the least committable unit would be for this patch. I wonder if it might be sufficient to provide a facility for streaming WAL, plus a standalone tool for receving it and storing it to a

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-09 Thread Simon Riggs
On Sat, 2009-08-08 at 22:02 -0400, Robert Haas wrote: I think it would also be fair to point out that you keep saying that you're going to deliver this patch for 8.5, but you haven't provided any real timetable as to when you're going to start working on it or when it'll be completed.

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-09 Thread Robert Haas
On Sun, Aug 9, 2009 at 6:11 AM, Simon Riggssi...@2ndquadrant.com wrote: I'm working on HS; I've said so clearly and say it again now. To my knowledge, no other Postgres project has committed to a timetable for delivery, so I'm not clear why you think one should have been given here, or why the

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-09 Thread Simon Riggs
On Sat, 2009-08-08 at 13:12 -0400, Bruce Momjian wrote: Simon Riggs wrote: I'm not sure why you're stirring this up again. You stated: - It's going to be very confusing if people submit their own versions of - it. So now we have mine, Heikki's and Robert's. I'd like this to stop -

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-09 Thread Bruce Momjian
Simon Riggs wrote: On Sat, 2009-08-08 at 13:12 -0400, Bruce Momjian wrote: Simon Riggs wrote: I'm not sure why you're stirring this up again. You stated: - It's going to be very confusing if people submit their own versions of - it. So now we have mine, Heikki's and

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-09 Thread Robert Haas
On Sun, Aug 9, 2009 at 2:43 PM, Simon Riggssi...@2ndquadrant.com wrote: I've said very clearly that I am working on this and it's fairly laughable to suggest that anybody thought I wasn't. What more should I do to prove something is active if you won't accept my clearly spoken word? How did

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-08 Thread Bruce Momjian
Robert Haas wrote: On Fri, Aug 7, 2009 at 11:33 PM, Bruce Momjianbr...@momjian.us wrote: Third, Robert, you should have communicated to the list that you were going to work on the patch, so that there would not be duplicate effort if someone else was also working on it. ?As I understood it,

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-08 Thread Robert Haas
On Fri, Aug 7, 2009 at 11:33 PM, Bruce Momjianbr...@momjian.us wrote: Third, Robert, you should have communicated to the list that you were going to work on the patch, so that there would not be duplicate effort if someone else was also working on it.  As I understood it, Heikki was in control

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-08 Thread Robert Haas
On Sat, Aug 8, 2009 at 12:02 AM, Bruce Momjianbr...@momjian.us wrote: Well, Simon stated that your version should now be used as the most recent one, so I would call that a success. Fair enough, but it still needs more work. I had some review comments I was hoping to get responses to, in the

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-08 Thread Simon Riggs
On Sat, 2009-08-08 at 00:02 -0400, Bruce Momjian wrote: Also, to my knowledge, nobody has really looked through the results to see if they are any good, so the success of the endeavor remains in doubt from my point of view. That's a bit of a shame because I am interested in putting some

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-08 Thread Bruce Momjian
Simon Riggs wrote: On Sat, 2009-08-08 at 00:02 -0400, Bruce Momjian wrote: Also, to my knowledge, nobody has really looked through the results to see if they are any good, so the success of the endeavor remains in doubt from my point of view. That's a bit of a shame because I am

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-08 Thread Robert Haas
On Sat, Aug 8, 2009 at 1:12 PM, Bruce Momjianbr...@momjian.us wrote: You are right you don't have to justify anything, but neither can you claim ownership of the patch/feature and complain that others are working on it too.  This is a community project --- if you want your patches to remain

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-08-07 Thread Bruce Momjian
Joshua D. Drake wrote: On Wed, 2009-07-15 at 17:27 +0100, Simon Riggs wrote: On Tue, 2009-07-14 at 21:12 -0400, Robert Haas wrote: It's going to be very confusing if people submit their own versions of it. So now we have mine, Heikki's and Robert's. I'd like this to stop please,

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-07-20 Thread Joshua D. Drake
On Wed, 2009-07-15 at 17:27 +0100, Simon Riggs wrote: On Tue, 2009-07-14 at 21:12 -0400, Robert Haas wrote: It's going to be very confusing if people submit their own versions of it. So now we have mine, Heikki's and Robert's. I'd like this to stop please, have a little faith and a little

Re: [HACKERS] hot standby - merged up to CVS HEAD

2009-07-15 Thread Simon Riggs
On Tue, 2009-07-14 at 21:12 -0400, Robert Haas wrote: 1. Downloaded norecoveryprocs-1.patch from http://archives.postgresql.org/message-id/49a64d73.6090...@enterprisedb.com http://archives.postgresql.org/message-id/4a4dbf8f.8040...@enterprisedb.com I have to confess that I had no idea that

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-03-03 Thread Simon Riggs
On Mon, 2009-03-02 at 21:11 -0500, Robert Treat wrote: On Wednesday 25 February 2009 16:43:54 Simon Riggs wrote: On Wed, 2009-02-25 at 13:33 -0800, Josh Berkus wrote: You raised that as an annoyance previously because it means that connection in hot standby mode may be delayed in cases

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-03-03 Thread Robert Treat
On Tuesday 03 March 2009 03:22:30 Simon Riggs wrote: On Mon, 2009-03-02 at 21:11 -0500, Robert Treat wrote: On Wednesday 25 February 2009 16:43:54 Simon Riggs wrote: On Wed, 2009-02-25 at 13:33 -0800, Josh Berkus wrote: You raised that as an annoyance previously because it means that

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-03-02 Thread Robert Treat
On Wednesday 25 February 2009 16:43:54 Simon Riggs wrote: On Wed, 2009-02-25 at 13:33 -0800, Josh Berkus wrote: You raised that as an annoyance previously because it means that connection in hot standby mode may be delayed in cases of heavy, repeated use of significant numbers of

Re: [HACKERS] Hot Standby - 8.5

2009-02-27 Thread Bruce Momjian
Heikki Linnakangas wrote: As discussed at http://archives.postgresql.org/message-id/603c8f070902251956s16eee4a7l495d75d3ddccc...@mail.gmail.com, it's time to stop pushing hot standby into 8.4, take the time to work out the remaining details, and schedule it for 8.5. It will be a great

Re: [HACKERS] Hot standby, recovery procs

2009-02-26 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-02-24 at 21:59 +0200, Heikki Linnakangas wrote: What benefit would we gain from separating them, especially since we now have working, tested code? Simplicity. That matters a lot. Removing the distinction between unobserved xids and already-observed running

Re: [HACKERS] Hot standby, recovery procs

2009-02-26 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-02-24 at 21:59 +0200, Heikki Linnakangas wrote: I think if I had not made those into procs you would have said that they are so similar it would aid code readability to have them be the same. And in fact I suggested earlier that we get rid of the unobserved xids

Re: [HACKERS] Hot standby, recovery procs

2009-02-26 Thread Simon Riggs
On Thu, 2009-02-26 at 10:04 +0200, Heikki Linnakangas wrote: we keep track of which xids have already been reported in the WAL (similar to what you had in an earlier version of the patch) You objected to doing exactly that earlier. Why is it OK to do it now that you are proposing it? You

Re: [HACKERS] Hot standby, recovery procs

2009-02-26 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-26 at 10:04 +0200, Heikki Linnakangas wrote: we keep track of which xids have already been reported in the WAL (similar to what you had in an earlier version of the patch) You objected to doing exactly that earlier. I'm talking about the xidMarkedInWAL

Re: [HACKERS] Hot standby, recovery procs

2009-02-26 Thread Simon Riggs
On Thu, 2009-02-26 at 11:36 +0200, Heikki Linnakangas wrote: You haven't even given a good reason to make these changes. Simplicity. You used that argument in January to explain why the coupling should be reduced and now the same argument to put it back again. We don't have time to make

Re: [HACKERS] Hot standby, recovery procs

2009-02-26 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-26 at 11:36 +0200, Heikki Linnakangas wrote: You haven't even given a good reason to make these changes. Simplicity. You used that argument in January to explain why the coupling should be reduced and now the same argument to put it back again. That was

Re: [HACKERS] Hot standby, recovery procs

2009-02-26 Thread Simon Riggs
On Thu, 2009-02-26 at 12:19 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-26 at 11:36 +0200, Heikki Linnakangas wrote: You haven't even given a good reason to make these changes. Simplicity. You used that argument in January to explain why the coupling should

Re: [HACKERS] Hot standby, recovery infra

2009-02-26 Thread Fujii Masao
Hi, On Fri, Jan 30, 2009 at 7:47 PM, Simon Riggs si...@2ndquadrant.com wrote: That whole area was something I was leaving until last, since immediate shutdown doesn't work either, even in HEAD. (Fujii-san and I discussed this before Christmas, briefly). This problem remains in current HEAD. I

Re: [HACKERS] Hot standby, recovery infra

2009-02-26 Thread Heikki Linnakangas
Fujii Masao wrote: On Fri, Jan 30, 2009 at 7:47 PM, Simon Riggs si...@2ndquadrant.com wrote: That whole area was something I was leaving until last, since immediate shutdown doesn't work either, even in HEAD. (Fujii-san and I discussed this before Christmas, briefly). This problem remains in

Re: [HACKERS] Hot standby, recovery infra

2009-02-26 Thread Fujii Masao
Hi, On Fri, Feb 27, 2009 at 3:38 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I think the real problem here is that pg_standby traps SIGQUIT. The startup process doesn't receive the SIGQUIT because it's in system(), and pg_standby doesn't propagate it to the startup

Re: [HACKERS] Hot standby, recovery infra

2009-02-26 Thread Simon Riggs
On Thu, 2009-02-26 at 20:38 +0200, Heikki Linnakangas wrote: I think we should simply remove the signal handler for SIGQUIT from pg_standby. If you do this, please make it release dependent so pg_standby behaves correctly for the release it is being used with. -- Simon Riggs

Re: [HACKERS] Hot standby, recovery infra

2009-02-26 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-26 at 20:38 +0200, Heikki Linnakangas wrote: I think we should simply remove the signal handler for SIGQUIT from pg_standby. If you do this, please make it release dependent so pg_standby behaves correctly for the release it is being used with. Hmm, I

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-02-25 Thread Simon Riggs
On Wed, 2009-02-25 at 22:39 +0200, Heikki Linnakangas wrote: When we take the snapshot of running transactions in the master, in GetRunningTransactionData(), it only includes top-level xids and those subxids that are in the subxid caches. Overflowed subxids are not included. Isn't that a

Re: [HACKERS] Hot standby, recovery procs

2009-02-25 Thread Simon Riggs
On Tue, 2009-02-24 at 23:41 +, Simon Riggs wrote: On Tue, 2009-02-24 at 22:29 +0200, Heikki Linnakangas wrote: overwrites subxids array, and will resurrect any already aborted subtransaction. Isn't XLByteLT(proc-lsn, lsn) always true, because 'lsn' is the lsn of the WAL record

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-02-25 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-02-25 at 22:39 +0200, Heikki Linnakangas wrote: When we take the snapshot of running transactions in the master, in GetRunningTransactionData(), it only includes top-level xids and those subxids that are in the subxid caches. Overflowed subxids are not

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-02-25 Thread Simon Riggs
On Wed, 2009-02-25 at 23:08 +0200, Heikki Linnakangas wrote: That is exactly the reason why we don't treat an overflowed snapshot as a valid starting point. We don't? I don't see anything stopping it. In GetRunningTransactionData() we explicitly set latestRunningXid to

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-02-25 Thread Josh Berkus
You raised that as an annoyance previously because it means that connection in hot standby mode may be delayed in cases of heavy, repeated use of significant numbers of subtransactions. While most users still don't use explicit subtransactions at all, wouldn't this also affect users who use

Re: [HACKERS] Hot standby, running xacts, subtransactions

2009-02-25 Thread Simon Riggs
On Wed, 2009-02-25 at 13:33 -0800, Josh Berkus wrote: You raised that as an annoyance previously because it means that connection in hot standby mode may be delayed in cases of heavy, repeated use of significant numbers of subtransactions. While most users still don't use explicit

Re: [HACKERS] Hot standby, recovery procs

2009-02-24 Thread Simon Riggs
On Tue, 2009-02-24 at 10:40 +0200, Heikki Linnakangas wrote: (back to reviewing the main hot standby patch at last) Why do we need recovery procs? AFAICS the only fields that we use are xid and the subxid cache. Now that we also have the unobserved xids array, why don't we use it to track

Re: [HACKERS] Hot standby, recovery procs

2009-02-24 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-02-24 at 10:40 +0200, Heikki Linnakangas wrote: (back to reviewing the main hot standby patch at last) Why do we need recovery procs? AFAICS the only fields that we use are xid and the subxid cache. Now that we also have the unobserved xids array, why don't we

Re: [HACKERS] Hot standby, recovery procs

2009-02-24 Thread Simon Riggs
On Tue, 2009-02-24 at 21:59 +0200, Heikki Linnakangas wrote: We only need the lsn atrribute because we when we take the snapshot of running xids, we don't write it to the WAL immediately, and a new transaction might begin after that. If we close that gap in the master, we don't need the

Re: [HACKERS] Hot standby, recovery procs

2009-02-24 Thread Heikki Linnakangas
Simon Riggs wrote: On Tue, 2009-02-24 at 21:59 +0200, Heikki Linnakangas wrote: We only need the lsn atrribute because we when we take the snapshot of running xids, we don't write it to the WAL immediately, and a new transaction might begin after that. If we close that gap in the master, we

Re: [HACKERS] Hot standby, recovery procs

2009-02-24 Thread Simon Riggs
On Tue, 2009-02-24 at 21:59 +0200, Heikki Linnakangas wrote: I think if I had not made those into procs you would have said that they are so similar it would aid code readability to have them be the same. And in fact I suggested earlier that we get rid of the unobserved xids array, and

Re: [HACKERS] Hot standby, recovery procs

2009-02-24 Thread Simon Riggs
On Tue, 2009-02-24 at 22:29 +0200, Heikki Linnakangas wrote: Oh, right... But we have the same problem with the subtransactions, don't we? This block: /* * If our state information is later for this proc, then * overwrite it. It's possible for a

Re: [HACKERS] Hot standby, recovery infra

2009-02-18 Thread Simon Riggs
On Mon, 2009-02-09 at 17:13 +0200, Heikki Linnakangas wrote: Attached is an updated patch that does that, and I've fixed all the other outstanding issues I listed earlier as well. Now I'm feeling again that this is in pretty good shape. UpdateMinRecoveryPoint() issues a DEBUG2 message even

Re: [HACKERS] Hot standby, recovery infra

2009-02-18 Thread Heikki Linnakangas
Simon Riggs wrote: On Mon, 2009-02-09 at 17:13 +0200, Heikki Linnakangas wrote: Attached is an updated patch that does that, and I've fixed all the other outstanding issues I listed earlier as well. Now I'm feeling again that this is in pretty good shape. UpdateMinRecoveryPoint() issues a

Re: [HACKERS] Hot standby, recovery infra

2009-02-18 Thread Simon Riggs
On Wed, 2009-02-18 at 14:26 +0200, Heikki Linnakangas wrote: The outer if should ensure that it isn't printed repeatedly on an idle system. Regrettably not. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Hot standby, recovery infra

2009-02-18 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-02-18 at 14:26 +0200, Heikki Linnakangas wrote: The outer if should ensure that it isn't printed repeatedly on an idle system. Regrettably not. Ok, committed. I fixed that and some comment changes. I also renamed IsRecoveryProcessingMode() to

Re: [HACKERS] Hot standby, recovery infra

2009-02-18 Thread Simon Riggs
On Wed, 2009-02-18 at 18:01 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-02-18 at 14:26 +0200, Heikki Linnakangas wrote: The outer if should ensure that it isn't printed repeatedly on an idle system. Regrettably not. Ok, committed. Cool. I fixed that

Re: [HACKERS] Hot Standby: subxid cache changes

2009-02-13 Thread Simon Riggs
On Thu, 2009-02-12 at 14:23 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-12 at 09:50 +0200, Heikki Linnakangas wrote: So far so good, but what about all the other callers of SubTransGetParent()? For example, XactLockTableWait will fail an assertion if asked to

Re: [HACKERS] Hot Standby: subxid cache changes

2009-02-13 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-12 at 14:23 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-12 at 09:50 +0200, Heikki Linnakangas wrote: So far so good, but what about all the other callers of SubTransGetParent()? For example, XactLockTableWait will fail an assertion

Re: [HACKERS] Hot Standby: subxid cache changes

2009-02-13 Thread Simon Riggs
On Fri, 2009-02-13 at 10:55 +0200, Heikki Linnakangas wrote: The logic is: if there is no lock table entry for that xid *and* it is not in progress *and* it is not in pg_subtrans, then it must have been an aborted subtransaction of a currently active xact or it has otherwise completed.

Re: [HACKERS] Hot Standby: subxid cache changes

2009-02-13 Thread Simon Riggs
On Thu, 2009-02-12 at 09:50 +0200, Heikki Linnakangas wrote: It occurs to me that we don't need this patch for hot standby if we abuse the main xid array (SnapshotData.xip) to store the unobserved xids instead of the subxid array. That one is always scanned in XidInMVCCSnapshot. I think

Re: [HACKERS] Hot Standby: subxid cache changes

2009-02-13 Thread Heikki Linnakangas
Simon Riggs wrote: I think my proposal still holds water, but I also think it is probably time to say OK, let's make this simpler and take the subxid tuning off line. Agreed. We would need to increase the max size of the xip array by 2*max_connections. So an increase of 80kB on normal

Re: [HACKERS] Hot Standby: subxid cache changes

2009-02-12 Thread Simon Riggs
On Thu, 2009-02-12 at 09:50 +0200, Heikki Linnakangas wrote: It occurs to me that we don't need this patch for hot standby if we abuse the main xid array (SnapshotData.xip) to store the unobserved xids instead of the subxid array. That one is always scanned in XidInMVCCSnapshot. I think

Re: [HACKERS] Hot Standby: subxid cache changes

2009-02-12 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-12 at 09:50 +0200, Heikki Linnakangas wrote: So far so good, but what about all the other callers of SubTransGetParent()? For example, XactLockTableWait will fail an assertion if asked to wait on a subtransaction which is then released. I agree that it

Re: [HACKERS] Hot standby, recovery infra

2009-02-06 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-05 at 21:54 +0200, Heikki Linnakangas wrote: - If you perform a fast shutdown while startup process is waiting for the restore command, startup process sometimes throws a FATAL error which leads escalates into an immediate shutdown. That leads to different

Re: [HACKERS] Hot standby, recovery infra

2009-02-06 Thread Simon Riggs
On Fri, 2009-02-06 at 10:06 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-05 at 21:54 +0200, Heikki Linnakangas wrote: - If you perform a fast shutdown while startup process is waiting for the restore command, startup process sometimes throws a FATAL error which

Re: [HACKERS] Hot standby, recovery infra

2009-02-05 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-05 at 09:28 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I would suggest that at end of recovery we write the last LSN to the control file, so if we crash recover then we will always end archive recovery at the same place again should we re-enter it.

Re: [HACKERS] Hot standby, recovery infra

2009-02-05 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-05 at 09:28 +0200, Heikki Linnakangas wrote: I've changed the way minRecoveryPoint is updated now anyway, so it no longer happens every XLogFileRead(). Care to elucidate? I got rid of minSafeStartPoint, advancing minRecoveryPoint instead. And it's advanced

Re: [HACKERS] Hot standby, recovery infra

2009-02-05 Thread Simon Riggs
On Thu, 2009-02-05 at 10:07 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-05 at 09:28 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I would suggest that at end of recovery we write the last LSN to the control file, so if we crash recover then we will always

Re: [HACKERS] Hot standby, recovery infra

2009-02-05 Thread Simon Riggs
On Thu, 2009-02-05 at 10:31 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-05 at 09:28 +0200, Heikki Linnakangas wrote: I've changed the way minRecoveryPoint is updated now anyway, so it no longer happens every XLogFileRead(). Care to elucidate? I got rid of

Re: [HACKERS] Hot standby, recovery infra

2009-02-05 Thread Simon Riggs
On Thu, 2009-02-05 at 09:26 +, Simon Riggs wrote: This change seems speculative and also against what has previously been agreed with Tom. If he chooses not to comment on your changes, that's up to him, but I don't think you should remove things quietly that have been put there through

Re: [HACKERS] Hot standby, recovery infra

2009-02-05 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2009-02-05 at 10:31 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2009-02-05 at 09:28 +0200, Heikki Linnakangas wrote: I got rid of minSafeStartPoint, advancing minRecoveryPoint instead. And it's advanced in XLogFlush instead of XLogFileRead. I'll post

Re: [HACKERS] Hot standby, recovery infra

2009-02-05 Thread Simon Riggs
On Thu, 2009-02-05 at 11:46 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: So we might end up flushing more often *and* we will be doing it potentially in the code path of other users. For example, imagine a database that fits completely in shared buffers. If we update at every

<    2   3   4   5   6   7   8   9   10   >