Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 4:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Well, I think the real hole is that turning archive_mode=on results in WAL never being deleted unless it's successfully archived. Hm, good point. And at least in principle you could have SR setups that don't care about having a backing WAL archive. But we might be able to handle that like this: wal_mode={standby|archive|crash} # or whatever wal_segments_always=integer # keep this many segments always, for SR - like current wal_keep_segments wal_segments_unarchived=integer # keep this many unarchived segments, -1 for infinite max_wal_senders=integer # same as now archive_command=string # same as now So we always retain wal_segments_always segments, but if we have trouble with archiving we'll retain up to wal_segments_archived. And when that limit is reached, what happens? Panic shutdown? Silently drop unarchived data? Neither one sounds very good. Silently drop unarchived data. I agree that isn't very good, but think about it this way: if archive_command is failing, then our log shipping slave is not going to work. But letting the disk fill up on the primary does not make it any better. It just makes the primary stop working, too. Obviously, all of this stuff needs to be monitored or you're playing with fire, but I don't think having a safety valve on the primary is a stupid idea. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Tom Lane wrote: Personally I agree with your objection to crash but not with the objection to standby. Maybe this would be appropriate: wal_mode = minimal | archive | hot_standby Ok, here's a patch implementing this proposal. It adds a new wal_mode setting, leaving archive_mode as it is. If you try to enable archive_mode when wal_mode is 'minimal', you get a warning and archive_mode is silently ignored. Likewise streaming replication connections are not allowed if wal_mode is 'minimal'. recovery_connections now does nothing in the master. A bit more bikeshedding before I commit this: * Should an invalid combination throw an ERROR and refuse to start, instead of just warning? * How about naming the parameter wal_level instead of wal_mode? That would better convey that the higher levels add stuff on top of the lower levels, instead of having different modes that are somehow mutually exclusive. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com diff --git a/doc/src/sgml/backup.sgml b/doc/src/sgml/backup.sgml index eb5765a..6c6a504 100644 --- a/doc/src/sgml/backup.sgml +++ b/doc/src/sgml/backup.sgml @@ -689,8 +689,7 @@ archive_command = 'test ! -f /mnt/server/archivedir/%f amp;amp; cp %p /mnt/ser /para para -When varnamearchive_mode/ is literaloff/ and xref -linkend=guc-max-wal-senders is zero some SQL commands +When varnamewal_mode/ is literalminimal/ some SQL commands are optimized to avoid WAL logging, as described in xref linkend=populate-pitr. If archiving or streaming replication were turned on during execution of one of these statements, WAL would not diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index c5692ba..63ca749 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -1353,6 +1353,43 @@ SET ENABLE_SEQSCAN TO OFF; titleSettings/title variablelist + varlistentry id=guc-wal-mode xreflabel=wal_mode + termvarnamewal_mode/varname (typeenum/type)/term + indexterm + primaryvarnamewal_mode/ configuration parameter/primary + /indexterm + listitem + para +varnamewal_mode/ determines how much information is written +to the WAL. The default value is literalminimal/, which writes +only minimal information needed to recover from a crash or immediate +shutdown. literalarchive/ adds logging required for WAL archiving, +and literalhot_standby/ further adds extra information about +running transactions required to run read-only queries on a standby +server. +This parameter can only be set at server start. + /para + para +In literalminimal/ mode, WAL-logging of some bulk operations, like +commandCREATE INDEX/, commandCLUSTER/ and commandCOPY/ on +a table that was created or truncated in the same transaction can be +safely skipped, which can make those operations much faster, but +minimal WAL does not contain enough information to reconstruct the +data from a base backup and the WAL logs, so at least +literalarchive/ level must be used to enable WAL archiving +(xref linkend=guc-archive-mode) and streaming replication. See +also xref linkend=populate-pitr. + /para + para +In literalhot_standby/ mode, the same information is logged as +in literalarchive/ mode, plus information needed to reconstruct +the status of running transactions from the WAL. To enable read-only +queries on a standby server, varnamewal_mode/ must be set to +literalhot_standby/ on the primary. + /para + /listitem + /varlistentry + varlistentry id=guc-fsync xreflabel=fsync indexterm primaryvarnamefsync/ configuration parameter/primary @@ -1726,7 +1763,9 @@ SET ENABLE_SEQSCAN TO OFF; varnamearchive_mode/ and varnamearchive_command/ are separate variables so that varnamearchive_command/ can be changed without leaving archiving mode. -This parameter can only be set at server start. +This parameter can only be set at server start. It is ignored +unless varnamewal_mode/ is set to literalarchive/ or +literalhot_standby/. /para /listitem /varlistentry @@ -1884,16 +1923,14 @@ SET ENABLE_SEQSCAN TO OFF; /indexterm listitem para -Parameter has two roles. During recovery, specifies whether or not -you can connect and run queries to enable xref linkend=hot-standby. -During normal running, specifies whether additional information is written -to WAL to allow recovery connections on a standby server that reads -WAL data generated by this server. The default value is +During recovery, specifies whether or not you can connect and run +queries to enable xref linkend=hot-standby. The default value is
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas wrote: On Fri, Apr 23, 2010 at 4:11 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Well, I think the real hole is that turning archive_mode=on results in WAL never being deleted unless it's successfully archived. Hm, good point. And at least in principle you could have SR setups that don't care about having a backing WAL archive. But we might be able to handle that like this: wal_mode={standby|archive|crash} # or whatever wal_segments_always=integer # keep this many segments always, for SR - like current wal_keep_segments wal_segments_unarchived=integer # keep this many unarchived segments, -1 for infinite max_wal_senders=integer # same as now archive_command=string# same as now So we always retain wal_segments_always segments, but if we have trouble with archiving we'll retain up to wal_segments_archived. And when that limit is reached, what happens? Panic shutdown? Silently drop unarchived data? Neither one sounds very good. Silently drop unarchived data. I agree that isn't very good, but think about it this way: if archive_command is failing, then our log shipping slave is not going to work. But letting the disk fill up on the primary does not make it any better. It just makes the primary stop working, too. Obviously, all of this stuff needs to be monitored or you're playing with fire, but I don't think having a safety valve on the primary is a stupid idea. hmm not sure I agree - you need to monitor diskspace usage in general on a system for obvious reasons. I think dealing with that kind of stuff is not really in our realm. We are a relational database and we need to guard the data, silently dropping data is imho not a good idea. Just picture the typical scenario of maintenance during night times on the standby done by a sysadmin with some batch jobs running on the master just generating enough WAL to exceed the limit that will just cause the sysadmin to call the DBA in. In general the question really is will people set this to something sensible or rather to an absurdly high value just to avoid that their replication will ever break - I guess people will do that later in critical environments... Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Mon, Apr 26, 2010 at 8:05 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Tom Lane wrote: Personally I agree with your objection to crash but not with the objection to standby. Maybe this would be appropriate: wal_mode = minimal | archive | hot_standby Ok, here's a patch implementing this proposal. It adds a new wal_mode setting, leaving archive_mode as it is. If you try to enable archive_mode when wal_mode is 'minimal', you get a warning and archive_mode is silently ignored. Likewise streaming replication connections are not allowed if wal_mode is 'minimal'. recovery_connections now does nothing in the master. A bit more bikeshedding before I commit this: * Should an invalid combination throw an ERROR and refuse to start, instead of just warning? I think so. Otherwise silent breakage is a real possibility. * How about naming the parameter wal_level instead of wal_mode? That would better convey that the higher levels add stuff on top of the lower levels, instead of having different modes that are somehow mutually exclusive. That works for me. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Mon, Apr 26, 2010 at 8:05 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: * How about naming the parameter wal_level instead of wal_mode? That would better convey that the higher levels add stuff on top of the lower levels, instead of having different modes that are somehow mutually exclusive. That works for me. What happens in the future if we have more options and they don't fall into a neat superset order? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Mon, Apr 26, 2010 at 10:23 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Apr 26, 2010 at 8:05 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: * How about naming the parameter wal_level instead of wal_mode? That would better convey that the higher levels add stuff on top of the lower levels, instead of having different modes that are somehow mutually exclusive. That works for me. What happens in the future if we have more options and they don't fall into a neat superset order? We'll decide on the appropriate solution based on whatever our needs are at that time? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Folks, (a) is this checked in yet? (b) should we delay Beta to test it?\ -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Mon, Apr 26, 2010 at 2:15 PM, Josh Berkus j...@agliodbs.com wrote: (a) is this checked in yet? No. (b) should we delay Beta to test it?\ I suspect it's going to be checked in pretty soon, so that may not be necessary. Not my call, though. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 19:33 -0400, Robert Haas wrote: Principle of obvious breakage. That is a good principle. It can be applied both ways here. Changing user interfaces (or indeed, anything) to very little obvious gain is a considerable annoyance to users. IIABDFI We need to be aware of the timing issues on the project. Changing something that has been the same for years is just annoying to existing users and makes upgrading to our brand new shiny software much harder than we ourselves would like that to be. But also, deferring solutions to user problems for vague reasons also needs to be avoided because waiting til next release moves the time to fix from about 6 months to about 18 months on average, which crosses patience threshold. So in general, I seek to speed up necessary change and slow down unnecessary change requests. I think we're improving on both. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Fujii Masao wrote: On Fri, Apr 23, 2010 at 1:04 AM, Robert Haas robertmh...@gmail.com wrote: One way we could fix this is use 2 bits rather than 1 for XLogStandbyInfoMode. One bit could indicate that either archive_mode=on or max_wal_senders0, and the second bit could indicate that recovery_connections=on. If the second bit is unset, we could emit the existing complaint: recovery connections cannot start because the recovery_connections parameter is disabled on the WAL source server If the other bit is unset, then we could instead complain: recovery connections cannot start because archive_mode=off and max_wal_senders=0 on the WAL source server If we don't want to use two bits there, it's hard to really describe all the possibilities in a reasonable number of characters. The only thing I can think of is to print a message and a hint: recovery_connections cannot start due to incorrect settings on the WAL source server HINT: make sure recovery_connections=on and either archive_mode=on or max_wal_senders0 I haven't checked whether the hint would be displayed in the log on the standby, but presumably we could make that be the case if it's not already. I think the first way is better because it gives the user more specific information about what they need to fix. Thinking about how each case might happen, since the default for recovery_connections is 'on', it seems that recovery_connections=off will likely only be an issue if the user has explicitly turned it off. The other case, where archive_mode=off and max_wal_senders=0, will likely only occur if someone takes a snapshot of the master without first setting up archiving or SR. Both of these will probably happen relatively rarely, but since we're burning a whole byte for XLogStandbyInfoMode (plus 3 more bytes of padding?), it seems like we might as well snag one more bit for clarity. Thoughts? I like the second choice since it's simpler and enough for me. But I have no objection to the first. When we encounter the error, we would need to not only change those parameter values but also take a fresh base backup and restart the standby using it. The description of this required procedure needs to be in the document or error message, I think. I quite liked Robert's proposal to add an explicit GUC to control what extra information is logged (http://archives.postgresql.org/pgsql-hackers/2010-04/msg00509.php). It is quite difficult to explain the current behavior, a simple explicit wal_mode GUC would be a lot simpler. It wouldn't add any extra steps to setting the system up, you currently need to set archive_mode='on' anyway to enable archiving. You would just set wal_mode='archive' or wal_mode='standby' instead, depending on what you want to do with the WAL. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 5:24 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: On Fri, Apr 23, 2010 at 1:04 AM, Robert Haas robertmh...@gmail.com wrote: One way we could fix this is use 2 bits rather than 1 for XLogStandbyInfoMode. One bit could indicate that either archive_mode=on or max_wal_senders0, and the second bit could indicate that recovery_connections=on. If the second bit is unset, we could emit the existing complaint: recovery connections cannot start because the recovery_connections parameter is disabled on the WAL source server If the other bit is unset, then we could instead complain: recovery connections cannot start because archive_mode=off and max_wal_senders=0 on the WAL source server If we don't want to use two bits there, it's hard to really describe all the possibilities in a reasonable number of characters. The only thing I can think of is to print a message and a hint: recovery_connections cannot start due to incorrect settings on the WAL source server HINT: make sure recovery_connections=on and either archive_mode=on or max_wal_senders0 I haven't checked whether the hint would be displayed in the log on the standby, but presumably we could make that be the case if it's not already. I think the first way is better because it gives the user more specific information about what they need to fix. Thinking about how each case might happen, since the default for recovery_connections is 'on', it seems that recovery_connections=off will likely only be an issue if the user has explicitly turned it off. The other case, where archive_mode=off and max_wal_senders=0, will likely only occur if someone takes a snapshot of the master without first setting up archiving or SR. Both of these will probably happen relatively rarely, but since we're burning a whole byte for XLogStandbyInfoMode (plus 3 more bytes of padding?), it seems like we might as well snag one more bit for clarity. Thoughts? I like the second choice since it's simpler and enough for me. But I have no objection to the first. When we encounter the error, we would need to not only change those parameter values but also take a fresh base backup and restart the standby using it. The description of this required procedure needs to be in the document or error message, I think. I quite liked Robert's proposal to add an explicit GUC to control what extra information is logged (http://archives.postgresql.org/pgsql-hackers/2010-04/msg00509.php). It is quite difficult to explain the current behavior, a simple explicit wal_mode GUC would be a lot simpler. It wouldn't add any extra steps to setting the system up, you currently need to set archive_mode='on' anyway to enable archiving. You would just set wal_mode='archive' or wal_mode='standby' instead, depending on what you want to do with the WAL. I liked it, too, but I sort of decided it didn't buy much. There are three separate sets of things that need to be controlled: 1. What WAL to emit - (a) just enough for crash recovery, (b) enough for log shipping, (c) enough for log shipping with recovery connections. 2. Whether to run the archiver. 3. Whether to allow streaming replication connections (and if so, how many). If the answer to (1) is just enough for crash recovery, then (2) and (3) must be no. But if (1) is either of the other two options, then any combination of answers for (2) and (3) is seemingly sensible, though having both (2) and (3) as no is probably of limited utility. But at a mimium, you could certainly have: crash recovery/no archiver/no SR log shipping/archiver/no SR log shipping/no archiver/SR log shipping/archiver/SR recovery connections/archiver/no SR recovery connections/no archiver/SR recovery connections/archiver/SR I don't see any reasonable way to package all of that up in a single GUC. Thoughts? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas wrote: On Fri, Apr 23, 2010 at 5:24 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I quite liked Robert's proposal to add an explicit GUC to control what extra information is logged (http://archives.postgresql.org/pgsql-hackers/2010-04/msg00509.php). It is quite difficult to explain the current behavior, a simple explicit wal_mode GUC would be a lot simpler. It wouldn't add any extra steps to setting the system up, you currently need to set archive_mode='on' anyway to enable archiving. You would just set wal_mode='archive' or wal_mode='standby' instead, depending on what you want to do with the WAL. I liked it, too, but I sort of decided it didn't buy much. There are three separate sets of things that need to be controlled: 1. What WAL to emit - (a) just enough for crash recovery, (b) enough for log shipping, (c) enough for log shipping with recovery connections. 2. Whether to run the archiver. 3. Whether to allow streaming replication connections (and if so, how many). Streaming replication needs the same information in the WAL as archiving does, there's no difference between 2 and 3. (the how many aspect of 3 is controlled by max_wal_senders). Let's have these three settings: wal_mode = crash/archive/standby (replaces archive_mode) archive_command max_wal_senders If wal_mode is set to 'crash', you can't set archive_command or max_wal_senders0. If it's set to 'archive', you can set archive_command and/or max_wal_senders for archiving and streaming replication, but the standby server won't allow queries. If you set it to 'standby', it will (assuming you've set recovery_connections=on in the standby). Note that wal_mode=standby replaces recovery_connections=on in the primary. I think this would be much easier to understand than the current situation. I'm not wedded to the GUC name or values, though, maybe it should be archive_mode=off/on/standby, or wal_mode=minimal/archive/full. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 7:12 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Robert Haas wrote: On Fri, Apr 23, 2010 at 5:24 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I quite liked Robert's proposal to add an explicit GUC to control what extra information is logged (http://archives.postgresql.org/pgsql-hackers/2010-04/msg00509.php). It is quite difficult to explain the current behavior, a simple explicit wal_mode GUC would be a lot simpler. It wouldn't add any extra steps to setting the system up, you currently need to set archive_mode='on' anyway to enable archiving. You would just set wal_mode='archive' or wal_mode='standby' instead, depending on what you want to do with the WAL. I liked it, too, but I sort of decided it didn't buy much. There are three separate sets of things that need to be controlled: 1. What WAL to emit - (a) just enough for crash recovery, (b) enough for log shipping, (c) enough for log shipping with recovery connections. 2. Whether to run the archiver. 3. Whether to allow streaming replication connections (and if so, how many). Streaming replication needs the same information in the WAL as archiving does, True. there's no difference between 2 and 3. (the how many aspect of 3 is controlled by max_wal_senders). False. I thought what you think too, but discovered otherwise when I read the code. Some uses of archive_mode are used to control what WAL is generated, but others control a *process* called the archiver. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas wrote: On Fri, Apr 23, 2010 at 7:12 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Robert Haas wrote: On Fri, Apr 23, 2010 at 5:24 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I quite liked Robert's proposal to add an explicit GUC to control what extra information is logged (http://archives.postgresql.org/pgsql-hackers/2010-04/msg00509.php). It is quite difficult to explain the current behavior, a simple explicit wal_mode GUC would be a lot simpler. It wouldn't add any extra steps to setting the system up, you currently need to set archive_mode='on' anyway to enable archiving. You would just set wal_mode='archive' or wal_mode='standby' instead, depending on what you want to do with the WAL. I liked it, too, but I sort of decided it didn't buy much. There are three separate sets of things that need to be controlled: 1. What WAL to emit - (a) just enough for crash recovery, (b) enough for log shipping, (c) enough for log shipping with recovery connections. 2. Whether to run the archiver. 3. Whether to allow streaming replication connections (and if so, how many). Streaming replication needs the same information in the WAL as archiving does, True. there's no difference between 2 and 3. (the how many aspect of 3 is controlled by max_wal_senders). False. I thought what you think too, but discovered otherwise when I read the code. Some uses of archive_mode are used to control what WAL is generated, but others control a *process* called the archiver. Hmm, never mind the archiver process, we could just launch it always and it would just sit idle if archive_command was not set. But a more serious concern is that if you set archive_mode=on, and archive_command='', we retain all WAL indefinitely, because it's not being archived, until you set archive_command to something that succeeds again. You're right, with the wal_mode='crash/archive/standby there would be no way to distinguish archiving is temporarily disabled, keep all accumulated WAL around and we're not archiving, but wal_mode='archive' to enable streaming replication. Ok, that brings us back to square one. We could still add the wal_mode GUC to explicitly control how much WAL is written (replacing recovery_connections in the primary), I think it would still make the system easier to explain. But it would add an extra hurdle to enabling archiving, you'd have to set wal_mode='archive', archive_mode='on', and archive_command. I'm not sure if that would be better or worse than the current situation. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Apr 23, 2010, at 13:12 , Heikki Linnakangas wrote: Let's have these three settings: wal_mode = crash/archive/standby (replaces archive_mode) archive_command max_wal_senders If wal_mode is set to 'crash', you can't set archive_command or max_wal_senders0. If it's set to 'archive', you can set archive_command and/or max_wal_senders for archiving and streaming replication, but the standby server won't allow queries. If you set it to 'standby', it will (assuming you've set recovery_connections=on in the standby). Note that wal_mode=standby replaces recovery_connections=on in the primary. I think this would be much easier to understand than the current situation. I'm not wedded to the GUC name or values, though, maybe it should be archive_mode=off/on/standby, or wal_mode=minimal/archive/full. Hm, but but that would preclude the possibility of running master and (log-shipping) slave off the same configuration, since one would need wal_mode=standby and the other recovery_connections=on. Whereas with the current GUCs, iarchive_mode=on, recovery_connections=on, archive_command=... should be a valid configuration for both master and slave, no? best regards, Florian Pflug smime.p7s Description: S/MIME cryptographic signature
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 7:40 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok, that brings us back to square one. We could still add the wal_mode GUC to explicitly control how much WAL is written (replacing recovery_connections in the primary), I think it would still make the system easier to explain. But it would add an extra hurdle to enabling archiving, you'd have to set wal_mode='archive', archive_mode='on', and archive_command. I'm not sure if that would be better or worse than the current situation. I wasn't either, that's why I gave up. It didn't seem worth doing a major GUC reorganization on the eve of beta unless there was a clear win. I think there may be a way to improve this but I don't think it's we should take the time now to figure out what it is. Let's revisit it for 9.1, and just improve the error reporting for now. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 8:54 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Apr 23, 2010 at 7:40 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok, that brings us back to square one. We could still add the wal_mode GUC to explicitly control how much WAL is written (replacing recovery_connections in the primary), I think it would still make the system easier to explain. But it would add an extra hurdle to enabling archiving, you'd have to set wal_mode='archive', archive_mode='on', and archive_command. I'm not sure if that would be better or worse than the current situation. I wasn't either, that's why I gave up. It didn't seem worth doing a major GUC reorganization on the eve of beta unless there was a clear win. I think there may be a way to improve this but I don't think it's we should take the time now to figure out what it is. Let's revisit it for 9.1, and just improve the error reporting for now. +1 Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:12 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Streaming replication needs the same information in the WAL as archiving does, True. FWIW, I still don't believe that claim, and I think it's complete folly to set the assumption in stone by choosing a user-visible GUC API that depends on it being true. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 07:54 -0400, Robert Haas wrote: Let's revisit it for 9.1, and just improve the error reporting for now. +1 -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 12:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:12 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Streaming replication needs the same information in the WAL as archiving does, True. FWIW, I still don't believe that claim, and I think it's complete folly to set the assumption in stone by choosing a user-visible GUC API that depends on it being true. Huh? We're clearly talking about two different things here, because that doesn't make any sense. Archiving and streaming replication are just two means of transporting WAL records from point A to point B. By definition, any two manners of moving a byte stream around are isomorphic and can't possibly affect what that byte stream does or does not need to contain. What affects the WAL that must be emitted is the purpose for which it is to be used. As to that, I believe everyone (including the code) is in agreement that a minimum amount of WAL is always needed for crash recovery, plus if we want to do archive recovery on another server there are some additional bits that must be emitted (XLogIsNeeded) and plus if further want to process queries on the standby then there are a few more bits beyond that (XLogStandbyInfoActive). ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 12:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: FWIW, I still don't believe that claim, and I think it's complete folly to set the assumption in stone by choosing a user-visible GUC API that depends on it being true. Huh? We're clearly talking about two different things here, because that doesn't make any sense. Archiving and streaming replication are just two means of transporting WAL records from point A to point B. Sorry, not enough caffeine. What I should have said was that Hot Standby could put stronger requirements on what gets put into WAL than archiving for recovery does. Heikki's proposal upthread was wal_mode='standby' versus wal_mode='archive' (versus 'off'), which seemed sensible to me. We realized some time ago that it was a good idea to separate archive_mode (what to put in WAL) from archive_command (whether we are actually archiving right now). If we fail to apply that same principle to Hot Standby, I think we'll come to regret it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Tom Lane wrote: We realized some time ago that it was a good idea to separate archive_mode (what to put in WAL) from archive_command (whether we are actually archiving right now). If we fail to apply that same principle to Hot Standby, I think we'll come to regret it. The recovery_connections GUC does that. If you enable it, the extra information required for hot standby is written to the WAL, otherwise it's not. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Tom Lane wrote: We realized some time ago that it was a good idea to separate archive_mode (what to put in WAL) from archive_command (whether we are actually archiving right now). If we fail to apply that same principle to Hot Standby, I think we'll come to regret it. The recovery_connections GUC does that. If you enable it, the extra information required for hot standby is written to the WAL, otherwise it's not. No, driving it off recovery_connections is exactly NOT that. It's confusing the transport mechanism with the desired WAL contents. I maintain that this design is exactly isomorphic to our original PITR GUC design wherein what got written to WAL was determined by the current state of archive_command. We eventually realized that was a bad idea. So is this. As a concrete example, there is nothing logically wrong with driving a hot standby slave from WAL records shipped via old-style pg_standby. Or how about wanting to turn off recovery_connections temporarily, but not wanting the archived WAL to be unable to support HS? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 2:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Tom Lane wrote: We realized some time ago that it was a good idea to separate archive_mode (what to put in WAL) from archive_command (whether we are actually archiving right now). If we fail to apply that same principle to Hot Standby, I think we'll come to regret it. The recovery_connections GUC does that. If you enable it, the extra information required for hot standby is written to the WAL, otherwise it's not. No, driving it off recovery_connections is exactly NOT that. It's confusing the transport mechanism with the desired WAL contents. I maintain that this design is exactly isomorphic to our original PITR GUC design wherein what got written to WAL was determined by the current state of archive_command. We eventually realized that was a bad idea. So is this. As a concrete example, there is nothing logically wrong with driving a hot standby slave from WAL records shipped via old-style pg_standby. Or how about wanting to turn off recovery_connections temporarily, but not wanting the archived WAL to be unable to support HS? You're all confused about what the different GUCs actually do. Which is probably not a good sign for their usability. But yeah, that's one of the things that concerned me, too. If you turn off max_wal_senders, it doesn't just make it so that no WAL senders can connect: it actually changes what gets WAL-logged. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Tom Lane t...@sss.pgh.pa.us wrote: As a concrete example, there is nothing logically wrong with driving a hot standby slave from WAL records shipped via old-style pg_standby. Or how about wanting to turn off recovery_connections temporarily, but not wanting the archived WAL to be unable to support HS? As one more concrete example, we are likely to find SR beneficial if it can feed into a warm standby, but only if we can also do traditional WAL file archiving from the same source at the same time. The extra logging for HS would be useless for us in any event. +1 for *not* tying WAL contents to the transport mechanism. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 13:45 -0400, Robert Haas wrote: Archiving and streaming replication are just two means of transporting WAL records from point A to point B. By definition, any two manners of moving a byte stream around are isomorphic and can't possibly affect what that byte stream does or does not need to contain. It is currently true, but there is no benefit in us constraining future implementation routes without good reason. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 2:43 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Tom Lane t...@sss.pgh.pa.us wrote: As a concrete example, there is nothing logically wrong with driving a hot standby slave from WAL records shipped via old-style pg_standby. Or how about wanting to turn off recovery_connections temporarily, but not wanting the archived WAL to be unable to support HS? As one more concrete example, we are likely to find SR beneficial if it can feed into a warm standby, but only if we can also do traditional WAL file archiving from the same source at the same time. The extra logging for HS would be useless for us in any event. +1 for *not* tying WAL contents to the transport mechanism. OK. Well, it's a shame we didn't get this settled last week when I first brought it up, but it's not too late to try to straighten it out if we have a consensus behind changing it, which it's starting to sound like we do. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 15:05 -0400, Robert Haas wrote: we have a consensus behind changing it, which it's starting to sound like we do. I think you misread the +1s from Masao and myself. Those confusing things are options and I want them to remain optional, not compressed into a potentially too simple model based upon how the world looks right now. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 3:11 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2010-04-23 at 15:05 -0400, Robert Haas wrote: we have a consensus behind changing it, which it's starting to sound like we do. I think you misread the +1s from Masao and myself. Those confusing things are options and I want them to remain optional, not compressed into a potentially too simple model based upon how the world looks right now. I didn't, but Heikki, Kevin and Tom seem to be on the other side, so we at least have to consider where to go with it. We're going to need a bunch of GUCs any way we slice it. The issue is whether there's a way to slice it that involves fewer AND and OR operators that have to be understood by users. I'm still unconvinced of our ability to come up with a solid design in the time we have, but I think it would make sense to listen to proposals people want to make. I poked some holes in Heikki's design from this morning (which was, more or less, my design from last week) but that doesn't mean they can't be plugged. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Simon Riggs si...@2ndquadrant.com writes: Those confusing things are options and I want them to remain optional, not compressed into a potentially too simple model based upon how the world looks right now. What are you arguing is too simple? What *I* think is too simple is what we have got now, namely a GUC that controls both the availability of replication connections and the contents of WAL. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: ... I'm still unconvinced of our ability to come up with a solid design in the time we have, but I think it would make sense to listen to proposals people want to make. I poked some holes in Heikki's design from this morning (which was, more or less, my design from last week) but that doesn't mean they can't be plugged. The only hole I saw poked was the one about how archive_mode is used to decide whether to start the archiver process. I think we could reasonably deal with that by starting the archiver iff wal_mode 'crash'. There's no point in archiving otherwise, and the overhead of an idle archiver is small enough that we can live with the corner cases where you're starting an archiver you don't really need. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 15:18 -0400, Robert Haas wrote: We're going to need a bunch of GUCs any way we slice it. The issue is whether there's a way to slice it that involves fewer AND and OR operators that have to be understood by users. So we're proposing adding parameters to simplify things for users? I don't think fiddling is going to improve things significantly from a usability perspective, especially at the last minute. I'm guessing this conversation has more to do with the situation that some very clever people have a little time on their hands after a long period of hard work. I see no problem that needs to be solved, not alongside this water cooler at least. Smells like beta time. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Simon Riggs si...@2ndquadrant.com wrote: So we're proposing adding parameters to simplify things for users? I think it's a matter of having parameters which do simple, clear things; rather than magically interacting to guess what the user wants. What do you want to log? How many connections to you want to allow for streaming it? What's your script for sending it in archive file format? Is archiving turned on at the moment? Let's have GUC for each question, rather than having to work backwards from what you want to which combination of GUC settings gets you to that, or at least as close as the magic interpretation allows. I don't think fiddling is going to improve things significantly from a usability perspective, especially at the last minute. If it involves changing the internal variables in a dangerous way, perhaps we should settle for whatever we have at the moment. If it's a matter of how they get set from the GUCs, that doesn't sound very risky to me. Perhaps there are combinations which were previously disallowed which would need to be tested, but are there any other risks? [ad hominem digression] Please, can we keep it to the merits? It sounds like there are several reasonable use-cases which could be handled by HS/SR except for how our GUCs are set up for it. Why limit the uses to a subset of where it can be useful? I'm extraordinarily busy right now, which is why my skimming of these threads didn't alert me to the problem sooner. For that I apologize. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Simon Riggs si...@2ndquadrant.com writes: So we're proposing adding parameters to simplify things for users? Not so much simplify as make understandable; although flexibility is a concern too. I'm guessing this conversation has more to do with the situation that some very clever people have a little time on their hands after a long period of hard work. I see no problem that needs to be solved, not alongside this water cooler at least. Smells like beta time. [ shrug... ] I'm just trying to learn from history and not repeat a previous mistake. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 3:34 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... I'm still unconvinced of our ability to come up with a solid design in the time we have, but I think it would make sense to listen to proposals people want to make. I poked some holes in Heikki's design from this morning (which was, more or less, my design from last week) but that doesn't mean they can't be plugged. The only hole I saw poked was the one about how archive_mode is used to decide whether to start the archiver process. I think we could reasonably deal with that by starting the archiver iff wal_mode 'crash'. There's no point in archiving otherwise, and the overhead of an idle archiver is small enough that we can live with the corner cases where you're starting an archiver you don't really need. Well, I think the real hole is that turning archive_mode=on results in WAL never being deleted unless it's successfully archived. But we might be able to handle that like this: wal_mode={standby|archive|crash} # or whatever wal_segments_always=integer # keep this many segments always, for SR - like current wal_keep_segments wal_segments_unarchived=integer # keep this many unarchived segments, -1 for infinite max_wal_senders=integer # same as now archive_command=string# same as now So we always retain wal_segments_always segments, but if we have trouble with archiving we'll retain up to wal_segments_archived. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 14:56 -0500, Kevin Grittner wrote: Simon Riggs si...@2ndquadrant.com wrote: So we're proposing adding parameters to simplify things for users? I think it's a matter of having parameters which do simple, clear things; rather than magically interacting to guess what the user wants. What do you want to log? How many connections to you want to allow for streaming it? What's your script for sending it in archive file format? Is archiving turned on at the moment? Let's have GUC for each question, rather than having to work backwards from what you want to which combination of GUC settings gets you to that, or at least as close as the magic interpretation allows. I've just committed a change to make Hot Standby depend only upon the setting recovery_connections = on on the master. That makes it clear that there is one lever, not lots of confusing ones. That might forestall further changes, because the correct way of doing this was already as simple as people wanted it to be. The previous requirement was actually a bug: the method of WAL delivery has nothing at all to do with Hot Standby (currently). Not intended to stop further debate, if people wish. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: ... I'm still unconvinced of our ability to come up with a solid design in the time we have, but I think it would make sense to listen to proposals people want to make. I poked some holes in Heikki's design from this morning (which was, more or less, my design from last week) but that doesn't mean they can't be plugged. The only hole I saw poked was the one about how archive_mode is used to decide whether to start the archiver process. I think we could reasonably deal with that by starting the archiver iff wal_mode 'crash'. There's no point in archiving otherwise, and the overhead of an idle archiver is small enough that we can live with the corner cases where you're starting an archiver you don't really need. Agreed, but a more serious hole is what I pointed out at http://archives.postgresql.org/message-id/4bd18722.3090...@enterprisedb.com. That is, if you do: wal_mode=standby archive_command='' max_wal_senders=5 That would be a valid configuration for enabling streaming replication without archiving (which is possible and reasonable if you set the new wal_keep_segments setting high enough). But as things stand, WAL segments would be readied for archiving (.ready files would be created), but they'e never archived and will accumulate indefinitely in the master. You could work around that with archive_command='/usr/bin/true', but that's not user-frienfly. So my proposal would be: wal_mode=crash/archive/standby archive_mode=on/off # if on, wal_mode must be = 'archive' archive_command='command' max_wal_senders=integer # if 0, wal_mode must be = 'archive' replication_connections is not needed on the master anymore; on the standby it enables/disables hot standby. It is ignored on the master, to allow the same configuration file to be used on master and standby. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: Well, I think the real hole is that turning archive_mode=on results in WAL never being deleted unless it's successfully archived. Hm, good point. And at least in principle you could have SR setups that don't care about having a backing WAL archive. But we might be able to handle that like this: wal_mode={standby|archive|crash} # or whatever wal_segments_always=integer # keep this many segments always, for SR - like current wal_keep_segments wal_segments_unarchived=integer # keep this many unarchived segments, -1 for infinite max_wal_senders=integer # same as now archive_command=string# same as now So we always retain wal_segments_always segments, but if we have trouble with archiving we'll retain up to wal_segments_archived. And when that limit is reached, what happens? Panic shutdown? Silently drop unarchived data? Neither one sounds very good. I think either you want your WAL archived or you don't. Archive if it's convenient doesn't sound like a useful operating mode. So maybe we do indeed need to keep archive_mode as a separate toggle. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 23:10 +0300, Heikki Linnakangas wrote: So my proposal would be: wal_mode=crash/archive/standby OK, I agree to change in this area. I definitely don't like the word crash, which may scare and confuse people. I don't think I would ever set any parameter to a word like crash since it isn't clear whether it allows that event or protects against it. Also, I don't like the word standby on its own, since that has already been used for Warm Standby for some time, which corresponds to the archive setting and is therefore confusing. How about something like wal_additional_info = none | archive | connect Then its easy to understand that things slow down when you request additional information in the WAL, and also clear that Hot Standby requires slightly more info on top of that. It's also clear that this has nothing at all to do with the delivery mechanism. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2010-04-23 at 23:10 +0300, Heikki Linnakangas wrote: So my proposal would be: wal_mode=crash/archive/standby I definitely don't like the word crash, which may scare and confuse people. I don't think I would ever set any parameter to a word like crash since it isn't clear whether it allows that event or protects against it. Also, I don't like the word standby on its own, since that has already been used for Warm Standby for some time, which corresponds to the archive setting and is therefore confusing. Good points, although recovery instead of crash would seem to cover that. How about something like wal_additional_info = none | archive | connect Then its easy to understand that things slow down when you request additional information in the WAL, and also clear that Hot Standby requires slightly more info on top of that. It's also clear that this has nothing at all to do with the delivery mechanism. Are we going to support running warm standby through SR? If so, connect seems confusing for the level to support hot standby. Perhaps live?: wal_mode=recovery/archive/live -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Simon Riggs si...@2ndquadrant.com writes: How about something like wal_additional_info = none | archive | connect connect seems like a completely inappropriate word here. It is not obviously related to HS slaves and it could be taken to refer to ordinary database connections (sessions). Personally I agree with your objection to crash but not with the objection to standby. Maybe this would be appropriate: wal_mode = minimal | archive | hot_standby regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 4:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: How about something like wal_additional_info = none | archive | connect connect seems like a completely inappropriate word here. It is not obviously related to HS slaves and it could be taken to refer to ordinary database connections (sessions). Personally I agree with your objection to crash but not with the objection to standby. Maybe this would be appropriate: wal_mode = minimal | archive | hot_standby I was thinking maybe log_shipping instead of archive, since we're conflating the technology (log shipping) with the technology used to implement it (archiving or streaming). Possible crash_recovery rather than just crash where you have mimimal. I don't love hot_standby either but it might be the least of evils. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 16:50 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: How about something like wal_additional_info = none | archive | connect connect seems like a completely inappropriate word here. It is not obviously related to HS slaves and it could be taken to refer to ordinary database connections (sessions). Personally I agree with your objection to crash but not with the objection to standby. Maybe this would be appropriate: wal_mode = minimal | archive | hot_standby Sounds good, I'll go for that. In my understanding this means that archive_mode does completely and the max_wal_senders does not affect WAL contents? Does that mean that wal_mode can be SIGHUP now? It would be good. I think this is how to do that: At the start of every WAL-avoiding operation we could take a copy of wal_mode for the server and store in MyProc-wal_mode. At transaction start we would set that to not set. We could then make pg_start_backup() wait for all transactions with wal_mode set to complete before we continue. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 17:29 -0400, Robert Haas wrote: Possible crash_recovery rather than just crash where you have mimimal. Minimal is good because it is a performance option also, which is an aspect crash_recovery does not convey. (Plus we use the word crash again, which is too scary to use) -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 4:10 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So my proposal would be: wal_mode=crash/archive/standby archive_mode=on/off # if on, wal_mode must be = 'archive' archive_command='command' max_wal_senders=integer # if 0, wal_mode must be = 'archive' As a general design comment, I think we should avoid still having an archive_mode GUC but having it do something different. If we're going to change the semantics, we should also change the name, maybe to archiving. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, 2010-04-23 at 17:43 -0400, Robert Haas wrote: On Fri, Apr 23, 2010 at 4:10 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So my proposal would be: wal_mode=crash/archive/standby archive_mode=on/off # if on, wal_mode must be = 'archive' archive_command='command' max_wal_senders=integer # if 0, wal_mode must be = 'archive' As a general design comment, I think we should avoid still having an archive_mode GUC but having it do something different. If we're going to change the semantics, we should also change the name, maybe to archiving. We don't need *both* wal_mode and archive_mode, since archive_mode exists only to ensure that full WAL is written even when archive_command = '' momentarily. Should do this wal_mode=crash/archive/standby archive_command='command' max_wal_senders=integer # if 0, wal_mode must be = 'archive' and make wal_mode SIGHUP -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 4:10 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So my proposal would be: wal_mode=crash/archive/standby archive_mode=on/off # if on, wal_mode must be = 'archive' archive_command='command' max_wal_senders=integer # if 0, wal_mode must be = 'archive' As a general design comment, I think we should avoid still having an archive_mode GUC but having it do something different. If we're going to change the semantics, we should also change the name, maybe to archiving. Agreed on the general point, but AFAICS that proposal keeps the meaning of archive_mode the same as it was. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Simon Riggs si...@2ndquadrant.com writes: In my understanding this means that archive_mode does completely and the max_wal_senders does not affect WAL contents? I think we'd concluded that we have to keep archive_mode as a separate boolean. (Or we could use Heikki's idea of a max number of unarchived segments to hold onto, but I maintain that there are only two useful values and so we might as well leave it as the existing boolean.) Does that mean that wal_mode can be SIGHUP now? It would be good. I think this is how to do that: At the start of every WAL-avoiding operation we could take a copy of wal_mode for the server and store in MyProc-wal_mode. At transaction start we would set that to not set. We could then make pg_start_backup() wait for all transactions with wal_mode set to complete before we continue. I think that there are probably more synchronization issues than that, and in any case now is not the time to be trying to implement that feature. Maybe we can make it work in 9.1. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Simon Riggs si...@2ndquadrant.com writes: We don't need *both* wal_mode and archive_mode, since archive_mode exists only to ensure that full WAL is written even when archive_command = '' momentarily. No, you missed the point of the upthread discussion: archive_mode controls whether to start the archiver *and whether to hold onto not-yet-archived segments*. We could maybe finesse the first point but it's much harder to deal with the latter. The only workable alternative I can see to keeping archive_mode is to tell people to set archive_command to something like /usr/bin/true ... which is not simpler, especially not on Windows. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 6:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 4:10 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So my proposal would be: wal_mode=crash/archive/standby archive_mode=on/off # if on, wal_mode must be = 'archive' archive_command='command' max_wal_senders=integer # if 0, wal_mode must be = 'archive' As a general design comment, I think we should avoid still having an archive_mode GUC but having it do something different. If we're going to change the semantics, we should also change the name, maybe to archiving. Agreed on the general point, but AFAICS that proposal keeps the meaning of archive_mode the same as it was. Well, clearly it doesn't. Someone who thinks they can simply turn archive_mode=on and set archive_command is going to be sadly disappointed. Before, archive_mode arguably switched the server between two modes, with a whole set of behaviors associated with it: type of WAL logging, whether the archive runs, number of WAL segments maintained. Under any of the proposals on the table (other than, just adjust the error message, which still seems tempting) it's new purview will be more limited. ...Robert ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 6:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: Agreed on the general point, but AFAICS that proposal keeps the meaning of archive_mode the same as it was. Well, clearly it doesn't. Someone who thinks they can simply turn archive_mode=on and set archive_command is going to be sadly disappointed. Well, there is another variable that they'll have to adjust as well, but ISTM that archive_mode still does what it did before, ie, determine whether we attempt to archive WAL segments. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 7:07 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 6:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: Agreed on the general point, but AFAICS that proposal keeps the meaning of archive_mode the same as it was. Well, clearly it doesn't. Someone who thinks they can simply turn archive_mode=on and set archive_command is going to be sadly disappointed. Well, there is another variable that they'll have to adjust as well, but ISTM that archive_mode still does what it did before, ie, determine whether we attempt to archive WAL segments. But it doesn't do EVERYTHING that it did before. Changing the name would make that a lot more clear. Of course I just work here. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:07 PM, Tom Lane t...@sss.pgh.pa.us wrote: Well, there is another variable that they'll have to adjust as well, but ISTM that archive_mode still does what it did before, ie, determine whether we attempt to archive WAL segments. But it doesn't do EVERYTHING that it did before. Changing the name would make that a lot more clear. Of course I just work here. I think from the user's point of view it does what it did before. The fact that the actual content of WAL changed was an implementation detail that users weren't aware of. Now that we have two interacting features that affect WAL contents, it's getting too hard to hide that from users --- but I see no need to rename archive_mode. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 7:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:07 PM, Tom Lane t...@sss.pgh.pa.us wrote: Well, there is another variable that they'll have to adjust as well, but ISTM that archive_mode still does what it did before, ie, determine whether we attempt to archive WAL segments. But it doesn't do EVERYTHING that it did before. Changing the name would make that a lot more clear. Of course I just work here. I think from the user's point of view it does what it did before. The fact that the actual content of WAL changed was an implementation detail that users weren't aware of. Now that we have two interacting features that affect WAL contents, it's getting too hard to hide that from users --- but I see no need to rename archive_mode. Well, when people use their same settings that they used for 8.4 and it doesn't work, you can field those reports... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think from the user's point of view it does what it did before. The fact that the actual content of WAL changed was an implementation detail that users weren't aware of. Now that we have two interacting features that affect WAL contents, it's getting too hard to hide that from users --- but I see no need to rename archive_mode. Well, when people use their same settings that they used for 8.4 and it doesn't work, you can field those reports... I would expect that they'll get an error message that makes it clear enough what to do ;-). In any case, changing the name is hardly going to fix things so that 8.4 settings will still work, so why are you giving that case as an argument for it? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 7:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think from the user's point of view it does what it did before. The fact that the actual content of WAL changed was an implementation detail that users weren't aware of. Now that we have two interacting features that affect WAL contents, it's getting too hard to hide that from users --- but I see no need to rename archive_mode. Well, when people use their same settings that they used for 8.4 and it doesn't work, you can field those reports... I would expect that they'll get an error message that makes it clear enough what to do ;-). In any case, changing the name is hardly going to fix things so that 8.4 settings will still work, so why are you giving that case as an argument for it? Principle of obvious breakage. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: I would expect that they'll get an error message that makes it clear enough what to do ;-). In any case, changing the name is hardly going to fix things so that 8.4 settings will still work, so why are you giving that case as an argument for it? Principle of obvious breakage. And? If we do it by adding the new variable while not renaming archive_mode, then I'd expect an 8.4 configuration to yield an error along the lines of ERROR: invalid combination of configuration parameters HINT: To turn on archive_mode, you must set wal_mode to archive or hot_standby. (precise wording open to debate, but clearly we can do at least this well) whereas if we rename archive_mode, it's unlikely we can do better than ERROR: unrecognized parameter archive_mode Do you really think the second one is going to make any user happier than the first? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 8:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Apr 23, 2010 at 7:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: I would expect that they'll get an error message that makes it clear enough what to do ;-). In any case, changing the name is hardly going to fix things so that 8.4 settings will still work, so why are you giving that case as an argument for it? Principle of obvious breakage. And? If we do it by adding the new variable while not renaming archive_mode, then I'd expect an 8.4 configuration to yield an error along the lines of ERROR: invalid combination of configuration parameters HINT: To turn on archive_mode, you must set wal_mode to archive or hot_standby. (precise wording open to debate, but clearly we can do at least this well) whereas if we rename archive_mode, it's unlikely we can do better than ERROR: unrecognized parameter archive_mode Do you really think the second one is going to make any user happier than the first? OK, good point. I overlooked the fact that we could cross-check the parameter settings on the master - I was imagining the error showing up on the standby. Guess I'm a little slow today... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: recovery_connections cannot start (was Re: [HACKERS] master in standby mode croaks)
On Fri, Apr 23, 2010 at 1:04 AM, Robert Haas robertmh...@gmail.com wrote: One way we could fix this is use 2 bits rather than 1 for XLogStandbyInfoMode. One bit could indicate that either archive_mode=on or max_wal_senders0, and the second bit could indicate that recovery_connections=on. If the second bit is unset, we could emit the existing complaint: recovery connections cannot start because the recovery_connections parameter is disabled on the WAL source server If the other bit is unset, then we could instead complain: recovery connections cannot start because archive_mode=off and max_wal_senders=0 on the WAL source server If we don't want to use two bits there, it's hard to really describe all the possibilities in a reasonable number of characters. The only thing I can think of is to print a message and a hint: recovery_connections cannot start due to incorrect settings on the WAL source server HINT: make sure recovery_connections=on and either archive_mode=on or max_wal_senders0 I haven't checked whether the hint would be displayed in the log on the standby, but presumably we could make that be the case if it's not already. I think the first way is better because it gives the user more specific information about what they need to fix. Thinking about how each case might happen, since the default for recovery_connections is 'on', it seems that recovery_connections=off will likely only be an issue if the user has explicitly turned it off. The other case, where archive_mode=off and max_wal_senders=0, will likely only occur if someone takes a snapshot of the master without first setting up archiving or SR. Both of these will probably happen relatively rarely, but since we're burning a whole byte for XLogStandbyInfoMode (plus 3 more bytes of padding?), it seems like we might as well snag one more bit for clarity. Thoughts? I like the second choice since it's simpler and enough for me. But I have no objection to the first. When we encounter the error, we would need to not only change those parameter values but also take a fresh base backup and restart the standby using it. The description of this required procedure needs to be in the document or error message, I think. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers