subject:"Re\: \[HACKERS\] unite recovery.conf and postgresql.conf"


On 12/14/2011 08:56 PM, Josh Berkus wrote:

So for streaming replication, will I need to have a standby.enabled
file, or will there be a parameter in postgresql.conf (or included
files) which controls whether or not that server is a standby, available?

In the best of all possible worlds, I'd really like to have a GUC which
100% controls whether or not the current server is a standby


You keep asking the hard questions.  Right now, I kind of like that it's 
possible to copy a postgresql.conf file from master to standby and just 
use it.  That should still be possible with the realignment into GUCs:


-"standby_mode = on":  Ignored unless you've started the server with 
standby.enabled, won't bother the master if you include it.
-"primary_conninfo":  This will look funny on the master showing it 
connecting to itself, but it will get ignored there too.


I was hoping to just copy over a base backup, "pg_ctl standby" creates 
the needed file and starts the server, and I'm done.  Isn't that the 
easiest replication "hello, world" possible here?  If you think there's 
an easier way here, please describe it more; I'm missing it so far.


Some settings will look a bit weird in the identical postgresql.conf in 
this case, but it think it can be made to work.  Now, eventually you 
will have to sort this out, but my normal principle here is that any 
issue deferred until after people have a working system is automatically 
easier for them to stomach.  Yes there's complexity, but people are 
facing it after the happy dance when the standby works for the first 
time.  The unavoidable bad situation happens if you promote a standby 
made this way.  Replicating more standbys from it won't work; you have 
to fix primary_conninfo at some point.  But once you're the master, you 
should be able to change primary_conninfo anytime--even if you SIGHUP to 
reload, it will now be ignored--so sorting that out doesn't even require 
a server restart.  [Problem of how exactly to define a GUC with those 
properties while also doing the right thing when you are a standby was 
noted then snuck by quietly]


If that is replaced with an edit to the postgresql.conf, that makes the 
bar for setting up a standby higher in my mind.  Now we have every 
clusterware product forced into the position pgpool already finds 
itself, where it needs to cope with making at least one change to that 
file.  I can see a middle ground position where you can have the 
standby.enabled file, but you can also set something in the 
postgresql.conf, but now we're back to conflict and order resolution 
madness.  [See:  "which of postgresql.conf and recovery.conf should be 
read first?"]


[Crazy talk begins here, but without further abuse of parenthetical 
brackets]


There is a route this way I wouldn't mind wandering down, but it circles 
back to one of the even bigger debates.  I would be perfectly happy 
fully embracing multiple configuration files for the server by default 
on every install.  Then the things that vary depending on current role 
can all be put into one place, with some expected splits along this 
line.  Put all the stuff related to standby configuration in one file; 
then tools can know "I can overwrite this whole file" and that will be true.


There's an obvious objection here that "having this crap in two files is 
the problem we're trying to eliminate!"  I would still see this as 
forward, because at the very minimum that split should refactor the 
replication and recovery target pieces into different files.  Different 
tools will want to feel they "own" them and can rewrite them, and making 
that easy should be a major goal.  Also, it will be possible to 
rearrange them if you'd like in whatever order makes sense, which you 
can't do now for the recovery.conf part.  You'd just be breaking tools 
that might expect the default split doing that; if you don't care, have 
at it.


Wandering any distance down that whole road surely stretches the premise 
of "simple migration procedure using include" too far to be true 
anymore.  I was thinking that for 9.2, it seems feasible to get much of 
this legacy stuff sorted better (from the perspective of the person 
focused on simple replication), and add some enabling features.  No 
recovery.conf, everything is a GUC, migration path isn't so bad, people 
get exposed to new concepts for include file organization.  I'd like to 
do some bigger reorganization too, but that seems too much change to 
shove all into one release.  There's a simple path from there that leads 
to both easier tools all around and SET PERSISTENT, and it comes with a 
pile of disruption so big I could throw in "standby controls are now 
100% GUC" for you plus a unicorn and it would slip right by unnoticed.  
That's a tough roadmap to sell unless those promised benefits are proven 
first though.  And I'm thinking a release doing all that is going to 
want to be named 10.0--and what I could really use is a nice, solid 9.2 
that do

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-12-14 Thread Josh Berkus

Greg,

> Put the stuff you used to insert into recovery.conf into postgresql.conf
> instead.  If you don't like that, use another file and include it with
> one of the multiple options for that--same migration option I already
> suggested.  Run "pg_ctl recovery"; under the hood that's actually
> creating standby.enabled instead of recovery.conf, but you don't have to
> know that.  You'd suggested renaming it to reflect its most common usage
> now, and I thought that was quite sensible.  It helps with the "things
> have changed, please drive carefully" feel too.

So for streaming replication, will I need to have a standby.enabled
file, or will there be a parameter in postgresql.conf (or included
files) which controls whether or not that server is a standby, available?

In the best of all possible worlds, I'd really like to have a GUC which
100% controls whether or not the current server is a standby.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf


On 12/14/2011 04:57 PM, Josh Berkus wrote:

How will things work for PITR?


Left that out mainly because I was already running too long there, but I 
do think there's a reasonable path.  There is one additional wrinkle in 
there to consider that I've thought of so far, falling into the "what is 
the best UI for this?" category I think.


Put the stuff you used to insert into recovery.conf into postgresql.conf 
instead.  If you don't like that, use another file and include it with 
one of the multiple options for that--same migration option I already 
suggested.  Run "pg_ctl recovery"; under the hood that's actually 
creating standby.enabled instead of recovery.conf, but you don't have to 
know that.  You'd suggested renaming it to reflect its most common usage 
now, and I thought that was quite sensible.  It helps with the "things 
have changed, please drive carefully" feel too.


It seems possible to have two files for state kickoff/tracking here 
instead, maybe have recovery.enabled and standby.enabled.  Is that extra 
complexity a useful thing?  I haven't dug into that new topic much yet.  
(Look at that!  I think I just found a *new* topic here!)


There are some questions around what to do when it's done.  The new 
proposed behavior is to delete the standby.enabled file.  But that 
doesn't remove the changes made for recovery like the old recovery.done 
rename did.  This is why I commented that some more thinking is likely 
needed about how to handle seeing those only-makes-sense-in-recovery 
options when not being started for recovery/standby; it's not obvious 
that any approach will make everyone happy.


If you want to do something special yourself to clean that up, there's 
already recovery_end_command available for that.  Let's say you wanted 
to force the old name and did "include_if_exists conf.d/recovery.conf", 
to trick it even if the patrolling for the name idea goes in.  Now you 
could do:


recovery_end_command = 'rm -f /tmp/pgsql.trigger.5432 && mv 
conf.d/recovery.conf conf.d/recovery.done'


Like some people are used to and might still prefer for some reason.  
There'd be time left over to head out to the lawn and yell at the kids 
there.  Actually, this might be the right approach for tools that are 
trying to change as little as possible but add quick 9.2 compatibility.


I think there's enough pluggable bits in every direction here that 
people can assemble the setup they'd like out of the available parts,


Maybe these slightly different semantics between archive recovery and 
standby mode are exactly why they should be kicked off by differently 
named files?


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf


On 12/14/2011 04:47 PM, Magnus Hagander wrote:

You say "the server can just delete it". But how will this work if the
file is *not* in the data directory? If you are on a Debian style
system for example, where all these files go in /etc/postgresql -
wouldn't that suddenly require the postgres user to have write access
in this directory? If it actually has to be the server that modifies
the file, I think it *does* make sense for this file to live in the
data directory...
   


Perhaps I should have softened the suggestion to "relocating the 
postgresql.conf makes it *possible* to have no user writable files in 
the data directory".  That was one of the later additions I made, it 
didn't bake overnight before sending like the bulk did.


A Debian system might want it to stay in the data directory.  If we 
consider this not normally touched by the user state information--they 
can poke it by hand, but the preferred way is to use pg_ctl--perhaps it 
could live in /var/run/postgresql instead.  [Surely I'll find out 
momentarily, now that I've trolled everyone here who is more familiar 
than me with the rules around what goes into /var]


I think the bigger idea I was trying to support in this part is just how 
many benefits there are from breaking this role into one decoupled from 
the main server configuration.  It's not a new suggestion, but I think 
it was cut down by backward compatibility concerns before being fully 
explored.  It seems all of the good ways to provide cleaner UIs need 
that, and it surely gives better flexibility to packagers for it to 
float free from the config.  Who can predict what people will end up 
doing in their packages.  (And the Gentoo changes have proven it's not 
just Debian)


If we drag this conversation back toward the best way to provide that 
cleaner UI, but can pick up enough agreement that backward compatibility 
limited to the sort of migration ideas I outlined is acceptable, I'd be 
happy with that progress.  Hopes of reaching that point is the reason I 
dumped time into those alternative include options.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-12-14 Thread Josh Berkus

On 12/14/11 1:16 PM, Greg Smith wrote:
> 
> -Creating a standby.enabled file in the directory that houses the
> postgresql.conf (same logic as "include" uses to locate things) puts the
> system into recovery mode.  That feature needs to save some state, and
> making those decisions based on existence of a file is already a thing
> we do.  Rather than emulating the rename to recovery.done that happens
> now, the server can just delete it, to keep from incorrectly returning
> to a state it's exited.  A UI along the lines of the promote one,
> allowing "pg_ctl standby", should fall out of here.  I think this is
> enough that people who just want to use replication features need never
> hear about this file at all, at least during the important to simplify
> first run through.

How will things work for PITR?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-12-14 Thread Magnus Hagander

On Wed, Dec 14, 2011 at 22:16, Greg Smith  wrote:
> I've submitted two patches for adding new include features to the
> postgresql.conf file.  While not quite commit quality yet, I hope everyone
> will agree their reviews this week suggest both are close enough that any
> number of people could finish them off.  Before re-opening this can of
> worms, I wanted people to be comfortable that we can expect them to be
> available as building blocks before 9.2 development is finished.  Both of
> those came out of requests from this unification thread, and they're a
> helpful part of what I'd like to propose.
>
> I don't see any path forward here that still expects the recovery.conf file
> to function as it used to.  The "make replication easy" crowd will
> eventually be larger than the pre-9.0 user base, if it isn't already.  And
> they clearly want no parts of that thing.  There's been over a year of
> arguing around how to cope with it that will satisfy everyone, so many
> messages I couldn't even read them all usefully in our archives and had to
> go here:
>
> http://postgresql.1045698.n5.nabble.com/recovery-conf-location-td2854644.html
> http://postgresql.1045698.n5.nabble.com/unite-recovery-conf-and-postgresql-conf-td4785717.html
>
> I don't think it's possible.  What I would propose is a specification based
> on forced failure if there's any sign of recovery.conf, combined with the
> simplest migration path we can design to ease upgrades from older versions.
>  I think we can make the transition easy enough.  Users and tool vendors can
> make relatively simple changes to support 9.2 without changing everything
> they're used to just yet--while still being very clear deprecation has
> arrived and they should reconsider their approach.  Only bug-compatible
> levels of backwards compatibility would make this transparent to them, and
> there's way too many issues to allow moving forward that way--a forward path
> that as far as I can see is desired by the majority of users, and just as
> importantly for all of the potential new users we're impacting with the
> current mess.
>
> There's another, perhaps under considered, concern I want to highlight as
> well.  Tom has repeatedly noted that one of the worst problems here would go
> away if the "existence means do recovery" nature of recovery.conf went
> elsewhere.  And we know some packagers want to separate the necessary to
> manipulate configuration files from the database directory, for permissions
> and management reasons.  As Heikki nicely put it (over a year ago), "You
> don't want to give monitoring tools that decide on failover write access to
> the data directory".  Any information that the user is supplying for the
> purpose of specifying things needs to be easy to relocate to a separate
> config file area, instead of treating it more like a control file in
> $PGDATA.  Some chatting this morning with Simon pointed out a second related
> concern there, which makes ideas like "specify the path to the recovery.conf
> file" infeasible.  The data_directory is itself a parameter, so anything
> tied to that or a new GUC means that config files specified there we would
> need two passes.  First identify the data directory, then go back again to
> read recovery.conf from somewhere else.  And nobody wants to wander that
> way.  If it doesn't fit cleanly into the existing postgresql.conf parsing,
> it's gotta go.
>
> Here's the rough outline of what I think would work here:
>
> -All settings move into the postgresql.conf.
>
> -As of 9.2, relocating the postgresql.conf means there are no user writable
> files needed in the data directory.
>
> -Creating a standby.enabled file in the directory that houses the
> postgresql.conf (same logic as "include" uses to locate things) puts the
> system into recovery mode.  That feature needs to save some state, and
> making those decisions based on existence of a file is already a thing we
> do.  Rather than emulating the rename to recovery.done that happens now, the
> server can just delete it, to keep from incorrectly returning to a state
> it's exited.  A UI along the lines of the promote one, allowing "pg_ctl
> standby", should fall out of here.  I think this is enough that people who
> just want to use replication features need never hear about this file at
> all, at least during the important to simplify first run through.

You say "the server can just delete it". But how will this work if the
file is *not* in the data directory? If you are on a Debian style
system for example, where all these files go in /etc/postgresql -
wouldn't that suddenly require the postgres user to have write access
in this directory? If it actually has to be the server that modifies
the file, I think it *does* make sense for this file to live in the
data directory...

[cutting lots of good explanations]

Other than that consideration, +1 for this proposal.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-12-14 Thread Robert Haas

On Wed, Dec 14, 2011 at 4:16 PM, Greg Smith  wrote:
> [ plan for deprecating recovery.conf ]

+1.  I'd be very happy with this plan.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

I've submitted two patches for adding new include features to the
postgresql.conf file. While not quite commit quality yet, I hope
everyone will agree their reviews this week suggest both are close
enough that any number of people could finish them off. Before
re-opening this can of worms, I wanted people to be comfortable that we
can expect them to be available as building blocks before 9.2
development is finished. Both of those came out of requests from this
unification thread, and they're a helpful part of what I'd like to propose.

I don't see any path forward here that still expects the recovery.conf
file to function as it used to. The "make replication easy" crowd will
eventually be larger than the pre-9.0 user base, if it isn't already.
And they clearly want no parts of that thing. There's been over a year
of arguing around how to cope with it that will satisfy everyone, so
many messages I couldn't even read them all usefully in our archives and
had to go here:

http://postgresql.1045698.n5.nabble.com/recovery-conf-location-td2854644.html
http://postgresql.1045698.n5.nabble.com/unite-recovery-conf-and-postgresql-conf-td4785717.html

I don't think it's possible. What I would propose is a specification
based on forced failure if there's any sign of recovery.conf, combined
with the simplest migration path we can design to ease upgrades from
older versions. I think we can make the transition easy enough. Users
and tool vendors can make relatively simple changes to support 9.2
without changing everything they're used to just yet--while still being
very clear deprecation has arrived and they should reconsider their
approach. Only bug-compatible levels of backwards compatibility would
make this transparent to them, and there's way too many issues to allow
moving forward that way--a forward path that as far as I can see is
desired by the majority of users, and just as importantly for all of the
potential new users we're impacting with the current mess.

There's another, perhaps under considered, concern I want to highlight
as well. Tom has repeatedly noted that one of the worst problems here
would go away if the "existence means do recovery" nature of
recovery.conf went elsewhere. And we know some packagers want to
separate the necessary to manipulate configuration files from the
database directory, for permissions and management reasons. As Heikki
nicely put it (over a year ago), "You don't want to give monitoring
tools that decide on failover write access to the data directory". Any
information that the user is supplying for the purpose of specifying
things needs to be easy to relocate to a separate config file area,
instead of treating it more like a control file in $PGDATA. Some
chatting this morning with Simon pointed out a second related concern
there, which makes ideas like "specify the path to the recovery.conf
file" infeasible. The data_directory is itself a parameter, so anything
tied to that or a new GUC means that config files specified there we
would need two passes. First identify the data directory, then go back
again to read recovery.conf from somewhere else. And nobody wants to
wander that way. If it doesn't fit cleanly into the existing
postgresql.conf parsing, it's gotta go.

Here's the rough outline of what I think would work here:

-All settings move into the postgresql.conf.

-As of 9.2, relocating the postgresql.conf means there are no user
writable files needed in the data directory.

-Creating a standby.enabled file in the directory that houses the
postgresql.conf (same logic as "include" uses to locate things) puts the
system into recovery mode. That feature needs to save some state, and
making those decisions based on existence of a file is already a thing
we do. Rather than emulating the rename to recovery.done that happens
now, the server can just delete it, to keep from incorrectly returning
to a state it's exited. A UI along the lines of the promote one,
allowing "pg_ctl standby", should fall out of here. I think this is
enough that people who just want to use replication features need never
hear about this file at all, at least during the important to simplify
first run through.

-If startup finds a recovery.conf file where it used to live at,
abort--someone is expecting the old behavior. Hint to RTFM or include a
short migration guide right on the spot. That can have a nice section
about how you might use the various postgresql.conf include* features if
they want to continue managing those files separately. Example: rename
it as replication.conf and use include_if_exists if you want to be able
to rename it to recovery.done like before. Or drop it into a conf.d
directory where the rename will make it then skipped.

-Tools such as pgpool that want to write a simple configuration file,
only touching the things that used to go into recovery.conf, can tell
people to do the same trick.

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-12-02 Thread Josh Berkus

All:

*ping*

Trying to restart this discussion, since the feature bogged down on
spec.  We have consensus that we need to change how replication mode is
mangaged; surely we can reach consensus on how to change it?

On 11/8/11 11:39 AM, Josh Berkus wrote:
> 
>> configuration data somewhere else, but we really need to be able to tell
>> the difference between "starting PITR", "continuing PITR after a
>> mid-recovery crash", and "finished PITR, up and running normally".
>> A GUC is not a good way to do that.
> 
> Does a GUC make sense to you for how to handle standby/master for
> replication?


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-15 Thread Robert Haas

On Mon, Nov 7, 2011 at 2:14 PM, Josh Berkus  wrote:
> 2. standby_mode becomes an ENUM: "off,standby,pitr".  It can be reset on
> server reload or through "pg_ctl promote"

I'm a little bit confused by the way we're dragging standby_mode into
this conversation.  If you're using pg_standby, you can set
standby_mode=off and still have a standby.  If you're using a simple
recovery command that just copies files, you need to set
standby_mode=on if you don't want the standby to exit recovery and
promote.  But I think of standby_mode as meaning "should we use the
internal standby loop rather than depending on an external tool?"
rather than "should we become a standby?".

Maybe I'm confused.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-08 Thread Josh Berkus


> configuration data somewhere else, but we really need to be able to tell
> the difference between "starting PITR", "continuing PITR after a
> mid-recovery crash", and "finished PITR, up and running normally".
> A GUC is not a good way to do that.

Does a GUC make sense to you for how to handle standby/master for
replication?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-07 Thread Tom Lane

Josh Berkus  writes:
> As I said elsewhere, I think that a modified version of Simon's proposal
> gets us all what we want except code cleanliness.  I'd like to hear from
> Tom on that issue.

Well, code complexity is hard to gauge without coding a draft
implementation, but I think this largely fails on UI complexity.
It's still paying too high a price for backwards compatibility IMO.

The more I read about this, the more doubtful I am that unifying PITR
recovery with standby mode is a good idea from the UI perspective.  They
may share a lot of common infrastructure but they need to be triggered
in fundamentally different ways.

In particular, one of the reasons that recovery.conf/.done was set up
the way that it was was to have a simple way of letting the system get
out of PITR mode; without that, a crash during PITR recovery is going
to lead to restarting from the PITR start point (because when we
restart, we find configuration settings telling us to do that).
We could possibly move the necessary state into pg_control, but keeping
it as a GUC is going to be a mess.  On the whole I still think a trigger
file is a sane design for that.  It may make sense to move the
configuration data somewhere else, but we really need to be able to tell
the difference between "starting PITR", "continuing PITR after a
mid-recovery crash", and "finished PITR, up and running normally".
A GUC is not a good way to do that.

The angst around this issue seems to me to largely stem from trying to
use a configuration setup designed for one-shot PITR scenarios for
hot standby scenarios, which are really pretty different.  We have to
divorce those two cases before we're going to have something that's
sane and usable ... and AFAICS that means giving up backwards
compatibility to some degree.

We got this wrong in 9.0, which everyone understood at the time was an
unpolished prototype implementation of replication.  I don't think it's
sensible to now move the goalposts and decree that we've got to be
fully backward compatible with our first-cut mistakes.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-07 Thread Josh Berkus


> Agreed.  This thread has already expended too much of our valuable time,
> in my opinion.

As I said elsewhere, I think that a modified version of Simon's proposal
gets us all what we want except code cleanliness.  I'd like to hear from
Tom on that issue.

The proposal is:

1. No changes are expected to PITR mode, unless required by the other
changes below.

2. standby_mode becomes an ENUM: "off,standby,pitr".  It can be reset on
server reload or through "pg_ctl promote"

3. we add a "recovery_file" filepath parameter to postgresql.conf.  This
points to the location of recovery.conf, if any, but does not error
fatally if the file doesn't exist, allowing recovery.conf/.done if we
want to support it.
3.a. we continue to support recovery.conf/.done trigger file
 behavior if recovery_file is set.  In this case,
 the recovery.conf file would contain the standby_mode GUC.
3.b. should pitr mode require recovery_file?  Doesn't seem
 like it.
3.c. we begin arguments on whether or not recovery_file
 should be set by default

4. Haas implements  SET PERSISTENT for postgresql.conf

5. pg_ctl promote uses SET PERSISTENT to change standby_mode so that
former standbys can be permanently promoted.

6. (optional) we add include_if_exists specifier for postgresql.conf,
allowing scripters to handle having a separate recovery.conf file in
other ways.

7. (optional) we add a "pg_ctl standby" command which starts up
postgres and sets standby_mode to "standby".  I'm not sure I see the
value in this though.

The above allows DBAs to operate in "backwards compatibility mode"
without forcing authors of new tools and scripts to abide by it.

Question: if both standby_mode and recovery_file are set, what should
happen if the recovery_file is not present?  For backwards
compatibility, we would use SET PERSISTENT to set standby_mode after the
server comes up.  Arguments?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-04 Thread Bruce Momjian

Josh Berkus wrote:
> > We can always do nothing, which is a safe and viable option.
> 
> Not really, no.  The whole recovery.conf thing is very broken and
> inhibits adoption of PostgreSQL because our users can't figure it out.
> 
> You've made it pretty clear that you're personally comfortable with how
> replication configuration works now, and aren't really interested in any
> changes.  That's certainly a valid viewpoint, but the users and
> contributors who find the API horribly unusable also have a valid
> viewpoint.  You don't automatically win arguments because you're on the
> side of backwards compatibility.
> 
> When we released binary replication in 9.0, I thought everyone knew that
> it was a first cut and that we'd be making some dramatic changes --
> including ones which broke things -- over the next few versions.  There
> was simply no way for us to know real user requirements until the
> feature was in the field and being deployed in production.  We would
> discover some things which really didn't work and that we had to break
> and remake.  And we have.
> 
> Now you are arguing for premature senescence, where our first API
> becomes our only API now and forever.  That's a road to project death.

Agreed.  This thread has already expended too much of our valuable time,
in my opinion.

I think we have enough agreement that we need a new API, so let's design
one.  If we can add some backward-compatibility here, great, but let's
not have that driving the discussion.  Replication is already complex
enough that having two ways to set this up just adds confusion. 
Replication/PITR does not affect SQL or applications --- it affects
admin scripts and tools, so they are just going to have to adjust.  

We are not going to make everyone happy, so let's just move forward ---
if people want to pout in the corner, I really don't care.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-03 Thread Greg Smith


On 09/24/2011 04:49 PM, Joshua Berkus wrote:
Well, we *did* actually come up with a reasonable way, but it died 
under an avalanche of bikeshedding and 
"we-must-do-everything-the-way-we-always-have-done". I refer, of 
course, to the "configuration directory" patch, which was a fine 
solution, and would indeed take care of the recovery.conf issues as 
well had we implemented it. We can *still* implement it, for 9.2.


That actually died from a lack of round-tuits, the consensus at the end 
of the bike-sheeding was pretty clear.  Last night I finally got 
motivated to fix the bit rot and feature set on that patch, to match 
what seemed to be the easiest path toward community approval.  One known 
bug left to resolve and I think it's ready to submit for the next CF.


I think includeifexists is also a good improvement, too, on a related 
arc to the main topic here.  If I can finish off the directory one (or 
get someone else to fix my bug) I should be able to follow up with that 
one.  The patches won't be that different.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-02 Thread Josh Berkus

Simon,

> Everyone has their own set of requirements. I've tried hard to fuse
> those together into a useful proposal, listening to all. Please bear
> in mind that I make my living in exactly the same way you do, so you
> must surely be aware I do this solely in the common interest.

Thank you for giving us a place to start. I have seen and commented on
your compromise.  Both Robert Treat and I poked holes in it.  It was a
good first attempt, but not a final attempt -- your compromise proposal
was heavily skewed towards maintaining the status quo at the expense of
improving functionality.  As a compromise, it was "70% Simon, 30%
everyone else".

> I don't force you to accept that proposal, but challenging it does
> require somebody to step up to the plate and work out a better
> detailed proposal rather than just restate what they personally want.

I made a proposal, which was based on modifying (and I believe,
improving) your proposal.

> We can always do nothing, which is a safe and viable option.

Not really, no.  The whole recovery.conf thing is very broken and
inhibits adoption of PostgreSQL because our users can't figure it out.

You've made it pretty clear that you're personally comfortable with how
replication configuration works now, and aren't really interested in any
changes.  That's certainly a valid viewpoint, but the users and
contributors who find the API horribly unusable also have a valid
viewpoint.  You don't automatically win arguments because you're on the
side of backwards compatibility.

When we released binary replication in 9.0, I thought everyone knew that
it was a first cut and that we'd be making some dramatic changes --
including ones which broke things -- over the next few versions.  There
was simply no way for us to know real user requirements until the
feature was in the field and being deployed in production.  We would
discover some things which really didn't work and that we had to break
and remake.  And we have.

Now you are arguing for premature senescence, where our first API
becomes our only API now and forever.  That's a road to project death.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Wed, Nov 2, 2011 at 6:48 PM, Josh Berkus  wrote:

> Well, as someone who sets up and admins replication for a bunch of
> clients, here's what I'd like to see:

Everyone has their own set of requirements. I've tried hard to fuse
those together into a useful proposal, listening to all. Please bear
in mind that I make my living in exactly the same way you do, so you
must surely be aware I do this solely in the common interest.

I don't force you to accept that proposal, but challenging it does
require somebody to step up to the plate and work out a better
detailed proposal rather than just restate what they personally want.
I don't rule out out that a better proposal exists and would be
incredibly happy if someone worked out the design, wrote it up and
desnaggled it.

We can always do nothing, which is a safe and viable option.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-02 Thread Robert Haas

On Wed, Nov 2, 2011 at 2:48 PM, Josh Berkus  wrote:
> Is anyone working on SET PERSISTENT?  I thought that got bike-shedded to
> death.

I think we had a fairly good sketch of how it could work mapped out,
mostly based around adding a postgresql.auto file.  I could dig up the
old discussions, if need be.

Basically, I'd be willing to get that implemented and bull it through
to completion in time for 9.2 if having that would make it easier for
everyone to accept the idea of turning the recovery.conf parameters
into GUCs.  Otherwise I probably will not get to it this cycle.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-02 Thread Josh Berkus

RH, Simon,

> I think that might have some possibilities.  But how does that work in
> detail?  If you set it to empty, then the recovery_* parameters are
> just GUCs, I suppose: which seems fine.  But if you set it to a
> non-empty value then what happens, exactly?  The recovery.conf
> settings clobber anything in postgresql.conf, and when we exit
> recovery we reload the config, discarding any settings we got from
> recovery.conf?  That might not be too bad.

Yeah, that's what I was picturing.  By tying backwards-compatibility to
a setting in pg.conf, you eliminate a lot of changes for a DBA
accidentally enabling it.  This also then supports re-locating the
recovery.conf file, which has been an issue for a long time.

> I think we need to back up and figure out what problem we're trying to
> solve here.  IMV, the answer is "setting up a standby is too
> complicated and requiring yet another configuration file to do it
> makes it even more complicated".   If the mechanism we introduce to
> "solve" that problem is more complicated than what we have now, it
> might end up being a net regression in terms of usability.

Well, as someone who sets up and admins replication for a bunch of
clients, here's what I'd like to see:

1. no more using a configuration file as a trigger
2. ability to put replication configuration in postgresql.conf or in a
manually designated include file
3. having replication configuration show up in pg_settings

The three settings above would make my life as a contract DBA much
easier ... and I presume help a lot of our users like me.  Among other
things, fixing the 3 things above would make replication integrate a lot
better with configuration management systems and monitoring.

> I feel like changing everything that's currently in recovery.conf to
> GUCs and implementing SET PERSISTENT would give everyone what they
> need, admittedly without perfect backward compatibility, but perhaps
> close enough for government work, and a step forward overall.

Is anyone working on SET PERSISTENT?  I thought that got bike-shedded to
death.

> So backwards compatibility is important for downstream software.
>

If it wasn't, we wouldn't be having this discussion.  However, backwards
compatibility is not necessarily the *most* important consideration.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Wed, Nov 2, 2011 at 1:11 AM, Josh Berkus  wrote:

> There is no way we're getting distro packagers to switch from pg_ctl
> start.  Also, a lot of distros use the "postgres" command rather than
> pg_ctl anything.

So backwards compatibility is important for downstream software.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Wed, Nov 2, 2011 at 1:11 AM, Josh Berkus  wrote:

> There is no way we're getting distro packagers to switch from pg_ctl
> start.  Also, a lot of distros use the "postgres" command rather than
> pg_ctl anything.

So backwards compatibility is important for downstream software.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Wed, Nov 2, 2011 at 1:45 AM, Robert Haas  wrote:

> I think that might have some possibilities.  But how does that work in
> detail?

My thoughts also. I want to see the detail on an alternate proposal so
we can decide things sensibly.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 9:11 PM, Josh Berkus  wrote:
> On 11/1/11 12:29 PM, Robert Treat wrote:
>> "Starting in 9.2, you should use pg_ctl standby to launch your
>> database for normal operations and/or in cases where you are writing
>> init scripts to control your production databases. For backwards
>> compatibility, if you require the old behavior of using a
>> recovery.conf, we would recommend you use pg_ctl start instead".
>
> Gah.
>
> There is no way we're getting distro packagers to switch from pg_ctl
> start.  Also, a lot of distros use the "postgres" command rather than
> pg_ctl anything.
>
> Messing with pg_ctl is not really a solution for this, since few people
> in production environments call it directly.  Nobody I know, anyway.
>
> So Simon's suggested compromise still puts backwards compatibility ahead
> of promoting the new API.  This would result in nobody supporting the
> new API until the day we remove the old one from the code.

Which will never happen, since part of the proposal is that PITR will
only be supported using the old method.

> I think adding "recovery_conf_location = ''" to postgresql.conf is a
> much better compromise.  Assuming we can stand the code complexity ...

I think that might have some possibilities.  But how does that work in
detail?  If you set it to empty, then the recovery_* parameters are
just GUCs, I suppose: which seems fine.  But if you set it to a
non-empty value then what happens, exactly?  The recovery.conf
settings clobber anything in postgresql.conf, and when we exit
recovery we reload the config, discarding any settings we got from
recovery.conf?  That might not be too bad.

I think we need to back up and figure out what problem we're trying to
solve here.  IMV, the answer is "setting up a standby is too
complicated and requiring yet another configuration file to do it
makes it even more complicated".   If the mechanism we introduce to
"solve" that problem is more complicated than what we have now, it
might end up being a net regression in terms of usability.

I feel like changing everything that's currently in recovery.conf to
GUCs and implementing SET PERSISTENT would give everyone what they
need, admittedly without perfect backward compatibility, but perhaps
close enough for government work, and a step forward overall.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-01 Thread Josh Berkus

On 11/1/11 12:29 PM, Robert Treat wrote:
> "Starting in 9.2, you should use pg_ctl standby to launch your
> database for normal operations and/or in cases where you are writing
> init scripts to control your production databases. For backwards
> compatibility, if you require the old behavior of using a
> recovery.conf, we would recommend you use pg_ctl start instead".

Gah.

There is no way we're getting distro packagers to switch from pg_ctl
start.  Also, a lot of distros use the "postgres" command rather than
pg_ctl anything.

Messing with pg_ctl is not really a solution for this, since few people
in production environments call it directly.  Nobody I know, anyway.

So Simon's suggested compromise still puts backwards compatibility ahead
of promoting the new API.  This would result in nobody supporting the
new API until the day we remove the old one from the code.

I think adding "recovery_conf_location = ''" to postgresql.conf is a
much better compromise.  Assuming we can stand the code complexity ...

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 5:11 PM, Joshua Berkus  wrote:

> So, we have four potential paths regarding recovery.conf:
>
> 1) Break backwards compatibility entirely, and stop supporting recovery.conf 
> as a trigger file at all.

Note that is exactly what I have suggested when using "standby" mode
from pg_ctl.

But you already know that, since you said "If it's possible to run a
replica without having a recovery.conf file,
then I'm fine with your solution", and I already confirmed back to you
that would be possible.

> 2) Offer backwards compatibility only if "recovery_conf='filename'" is set in 
> postgresql.conf, then behave like Simon's compromise.
>
> 3) Simon's compromise.

See above. Calling it a compromise in this way implies nobody has been
given exactly what they ask for, but that is not the case.

> 4) Don't ever change how recovery.conf works.
>
> The only two of the above I see as being real options are (1) and (2).  (3) 
> would, as Robert points out, cause DBAs to have unpleasant surprises when 
> some third-party tool creates a recovery.conf they weren't expecting. So:

Please read my proposal again. I'll be happy to answer questions if
you have any.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-01 Thread Robert Treat

On Tue, Nov 1, 2011 at 1:34 PM, Simon Riggs  wrote:
> On Tue, Nov 1, 2011 at 5:11 PM, Joshua Berkus  wrote:
>
>> So, we have four potential paths regarding recovery.conf:
>>
>> 1) Break backwards compatibility entirely, and stop supporting recovery.conf 
>> as a trigger file at all.
>
> Note that is exactly what I have suggested when using "standby" mode
> from pg_ctl.
>
> But you already know that, since you said "If it's possible to run a
> replica without having a recovery.conf file,
> then I'm fine with your solution", and I already confirmed back to you
> that would be possible.
>

"It's possible to run a replica without having a recovery.conf file"
is not the same thing as "If someone makes a recovery.conf file, it
won't break my operations". AIUI, you are not supporting the latter.


Robert Treat
conjecture: xzilla.net
consulting: omniti.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-01 Thread Robert Treat

On Tue, Nov 1, 2011 at 2:34 PM, Simon Riggs  wrote:
> On Tue, Nov 1, 2011 at 6:12 PM, Robert Treat  wrote:
>> On Tue, Nov 1, 2011 at 1:34 PM, Simon Riggs  wrote:
>>> On Tue, Nov 1, 2011 at 5:11 PM, Joshua Berkus  wrote:
>>>
 So, we have four potential paths regarding recovery.conf:

 1) Break backwards compatibility entirely, and stop supporting 
 recovery.conf as a trigger file at all.
>>>
>>> Note that is exactly what I have suggested when using "standby" mode
>>> from pg_ctl.
>>>
>>> But you already know that, since you said "If it's possible to run a
>>> replica without having a recovery.conf file,
>>> then I'm fine with your solution", and I already confirmed back to you
>>> that would be possible.
>>>
>>
>> "It's possible to run a replica without having a recovery.conf file"
>> is not the same thing as "If someone makes a recovery.conf file, it
>> won't break my operations". AIUI, you are not supporting the latter.
>
> Yes, that is part of the "combined proposal", which allows both old
> and new APIs.
>
> New API
>
> pg_ctl standby
>    will startup a server in standby mode, do not implicitly include
> recovery.conf and disallow recovery_target parameters in
> postgresql.conf
>    (you may, if you wish, explicitly have "include recovery.conf" in
> your postgresql.conf, should you desire that)
>
> Old API
>
> pg_ctl start
> and a recovery.conf has been created
>   will startup a server in PITR and/or replication, recovery.conf
> will be read automatically (as now)
>   so the presence of the recovery.conf acts as a trigger, only if we
> issue "start"
>
> So the existing syntax works exactly as now, but a new API has been
> created to simplify things in exactly the way requested. The old and
> the new API don't mix, so there is no confusion between them.
>
> You must still use the old API when you wish to perform a PITR, as
> explained by me, following comments by Peter.
>

Ah, thanks for clarifying, your earlier proposal didn't read that way
to me. It still doesn't solve the problem for tool makers of needing
to be able to deal with two possible implementation methods, but it
should be easier for them to make a choice. One thing though, I think
it would be better to have this work the other way around. ISTM we're
going to end up telling people to avoid using pg_ctl start and instead
use pg_ctl standby, which doesn't feel like the right answer. Ie.
"Starting in 9.2, you should use pg_ctl standby to launch your
database for normal operations and/or in cases where you are writing
init scripts to control your production databases. For backwards
compatibility, if you require the old behavior of using a
recovery.conf, we would recommend you use pg_ctl start instead".

Robert Treat
conjecture: xzilla.net
consulting: omniti.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-01 Thread Josh Berkus

On 11/1/11 10:34 AM, Simon Riggs wrote:
> On Tue, Nov 1, 2011 at 5:11 PM, Joshua Berkus  wrote:
> 
>> So, we have four potential paths regarding recovery.conf:
>>
>> 1) Break backwards compatibility entirely, and stop supporting recovery.conf 
>> as a trigger file at all.
> 
> Note that is exactly what I have suggested when using "standby" mode
> from pg_ctl.

I wasn't clear on that from the description of your proposal.  So are
you suggesting that, if we start postgresql with "pg_ctl standby" then
recovery.conf would not behave as a trigger file?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-11-01 Thread Joshua Berkus

Robert,

> In most cases we either break backwards compatibility or require some
> type of switch to turn on backwards compatibility for those who want
> it. While the above plan tries to do one better, it leaves me feeling
> that the thing I don't like about this is that it sounds like you are
> forcing backwards compatibility on people who would much rather just
> do things the new way. Given that, I foresee a whole new generation
> of
> confused users who end up setting their configs one way only to have
> someone else set the same config in the other file, or some tool dump
> out some config file, overriding what was really intended. This will
> also make things *harder* for those tool providers you are trying to
> help, as they will be forced to support the behavior *both ways*. I'd
> much rather see some type of switch which turns on the old behavior
> for those who really want it, because while you can teach the new
> behavior, if you can't prevent the old behavior, you're creating
> operational headaches for yourself.

This is a good point.  There's also the second drawback, which is complexity of 
code, which I believe that Tom Lane has brought up before; having two 
separate-but-equal paths for configuration is liable to lead to a lot of bugs.

So, we have four potential paths regarding recovery.conf:

1) Break backwards compatibility entirely, and stop supporting recovery.conf as 
a trigger file at all.

2) Offer backwards compatibility only if "recovery_conf='filename'" is set in 
postgresql.conf, then behave like Simon's compromise.

3) Simon's compromise.

4) Don't ever change how recovery.conf works.

The only two of the above I see as being real options are (1) and (2).  (3) 
would, as Robert points out, cause DBAs to have unpleasant surprises when some 
third-party tool creates a recovery.conf they weren't expecting. So:

(1) pros:
   * new, clean API
   * makes everyone update their tools
   * no confusion on "how to do failover"
   * code simplicity
 cons:
   * breaks a bunch of 3rd-party tools
   * or forces them to maintain separate 9.1 and 9.2 branches

(2) pros:
   * allows people to use only new API if they want
   * allows gradual update of tools
   * can also lump in relocatable recovery.conf as feature
  cons:
   * puts off the day when vendors pay attention to the new API
 (and even more kicking & screaming when that day comes)
   * confusion about "how to do failover"
   * code complexity

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 6:12 PM, Robert Treat  wrote:
> On Tue, Nov 1, 2011 at 1:34 PM, Simon Riggs  wrote:
>> On Tue, Nov 1, 2011 at 5:11 PM, Joshua Berkus  wrote:
>>
>>> So, we have four potential paths regarding recovery.conf:
>>>
>>> 1) Break backwards compatibility entirely, and stop supporting 
>>> recovery.conf as a trigger file at all.
>>
>> Note that is exactly what I have suggested when using "standby" mode
>> from pg_ctl.
>>
>> But you already know that, since you said "If it's possible to run a
>> replica without having a recovery.conf file,
>> then I'm fine with your solution", and I already confirmed back to you
>> that would be possible.
>>
>
> "It's possible to run a replica without having a recovery.conf file"
> is not the same thing as "If someone makes a recovery.conf file, it
> won't break my operations". AIUI, you are not supporting the latter.

Yes, that is part of the "combined proposal", which allows both old
and new APIs.

New API

pg_ctl standby
will startup a server in standby mode, do not implicitly include
recovery.conf and disallow recovery_target parameters in
postgresql.conf
(you may, if you wish, explicitly have "include recovery.conf" in
your postgresql.conf, should you desire that)

Old API

pg_ctl start
and a recovery.conf has been created
   will startup a server in PITR and/or replication, recovery.conf
will be read automatically (as now)
   so the presence of the recovery.conf acts as a trigger, only if we
issue "start"

So the existing syntax works exactly as now, but a new API has been
created to simplify things in exactly the way requested. The old and
the new API don't mix, so there is no confusion between them.

You must still use the old API when you wish to perform a PITR, as
explained by me, following comments by Peter.

There is no significant additional code or complexity required to do
this, but it adds considerable usefulness.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 6:36 PM, Josh Berkus  wrote:
> On 11/1/11 10:34 AM, Simon Riggs wrote:
>> On Tue, Nov 1, 2011 at 5:11 PM, Joshua Berkus  wrote:
>>
>>> So, we have four potential paths regarding recovery.conf:
>>>
>>> 1) Break backwards compatibility entirely, and stop supporting 
>>> recovery.conf as a trigger file at all.
>>
>> Note that is exactly what I have suggested when using "standby" mode
>> from pg_ctl.
>
> I wasn't clear on that from the description of your proposal.  So are
> you suggesting that, if we start postgresql with "pg_ctl standby" then
> recovery.conf would not behave as a trigger file?

Yes

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 2:04 PM, Robert Haas  wrote:
> On Tue, Nov 1, 2011 at 9:45 AM, Simon Riggs  wrote:
>> Why do we have this log message then, if it is OK to ignore changes
>> that have no effect?
>>
>> LOG:  parameter "shared_buffers" cannot be changed without restarting the 
>> server
>
> I believe we're logging the fact that we were unable to make the
> change, not the fact that it didn't have any effect.  Certainly, it
> *would* have an effect, if we were able to make it.  But we can't,
> without a restart, so we tell that to the user.
>
> But, for example, the hot_standby GUC - which already exists - does
> not do anything in normal running.  We don't need to (and don't)
> complain if the user tries to change the value in normal running,
> though: they're presumably just hoping it will take effect the next
> time they start up, which it will.

If there is precedence then we should follow it. So no error messages.

The only reason this point has any importance is that Fujii is
suggesting the code will be much more complicated if we retain
recovery.conf compatibility, since we need to mark GUCs with a flag as
to whether they can take effect during recovery. I don't think that's
a lot of work to add, and would make the code clearer than it is now.


> And the autovacuum parameter does
> not do anything during hot standby, but we don't need to (and don't)
> complain if the user changes it them; it just takes effect when we
> enter normal running.

That one needs to be set on a standby, in case it becomes a primary.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 1:23 PM, Robert Haas  wrote:
> On Tue, Nov 1, 2011 at 8:14 AM, Simon Riggs  wrote:
>> On Tue, Nov 1, 2011 at 12:06 PM, Robert Haas  wrote:
>>> On Tue, Nov 1, 2011 at 7:46 AM, Simon Riggs  wrote:
 If you change a parameter that only has effect during recovery then
 must get an error if it is changed during normal running.
>>>
>>> I don't see why.  If you're in normal running and someone changes a
>>> parameter that is irrelevant during normal running, that should be a
>>> no-op, not an error.
>>
>> How will it be made into a no-op, except by having a specific flag to
>> show that it is irrelevant during normal running?
>
> By default, changing a GUC just updates the value of some global
> variable inside every backend.  But unless there's some code that
> makes use of that global variable for some purpose, it doesn't have
> any practical effect.  Apart from whatever complexities may be imposed
> by our choice of implementation, I don't see how this would be any
> different from setting maintenance_work_mem in a particular session
> and then not running any CREATE INDEX or VACUUM commands in that
> session.

Why do we have this log message then, if it is OK to ignore changes
that have no effect?

LOG:  parameter "shared_buffers" cannot be changed without restarting the server

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 9:45 AM, Simon Riggs  wrote:
> Why do we have this log message then, if it is OK to ignore changes
> that have no effect?
>
> LOG:  parameter "shared_buffers" cannot be changed without restarting the 
> server

I believe we're logging the fact that we were unable to make the
change, not the fact that it didn't have any effect.  Certainly, it
*would* have an effect, if we were able to make it.  But we can't,
without a restart, so we tell that to the user.

But, for example, the hot_standby GUC - which already exists - does
not do anything in normal running.  We don't need to (and don't)
complain if the user tries to change the value in normal running,
though: they're presumably just hoping it will take effect the next
time they start up, which it will.  And the autovacuum parameter does
not do anything during hot standby, but we don't need to (and don't)
complain if the user changes it them; it just takes effect when we
enter normal running.

The only time I think we need to complain about an effort to change a
GUC is when the GUC won't take effect as soon as the user might
normally expect.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 8:14 AM, Simon Riggs  wrote:
> On Tue, Nov 1, 2011 at 12:06 PM, Robert Haas  wrote:
>> On Tue, Nov 1, 2011 at 7:46 AM, Simon Riggs  wrote:
>>> If you change a parameter that only has effect during recovery then
>>> must get an error if it is changed during normal running.
>>
>> I don't see why.  If you're in normal running and someone changes a
>> parameter that is irrelevant during normal running, that should be a
>> no-op, not an error.
>
> How will it be made into a no-op, except by having a specific flag to
> show that it is irrelevant during normal running?

By default, changing a GUC just updates the value of some global
variable inside every backend.  But unless there's some code that
makes use of that global variable for some purpose, it doesn't have
any practical effect.  Apart from whatever complexities may be imposed
by our choice of implementation, I don't see how this would be any
different from setting maintenance_work_mem in a particular session
and then not running any CREATE INDEX or VACUUM commands in that
session.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 12:06 PM, Robert Haas  wrote:
> On Tue, Nov 1, 2011 at 7:46 AM, Simon Riggs  wrote:
>> If you change a parameter that only has effect during recovery then
>> must get an error if it is changed during normal running.
>
> I don't see why.  If you're in normal running and someone changes a
> parameter that is irrelevant during normal running, that should be a
> no-op, not an error.

How will it be made into a no-op, except by having a specific flag to
show that it is irrelevant during normal running?

Fujii is saying we only need to mark GUCs if we keep recovery.conf. I
am saying we need to mark them whatever we do elsewhere.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 7:46 AM, Simon Riggs  wrote:
> If you change a parameter that only has effect during recovery then
> must get an error if it is changed during normal running.

I don't see why.  If you're in normal running and someone changes a
parameter that is irrelevant during normal running, that should be a
no-op, not an error.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

On Tue, Nov 1, 2011 at 5:59 AM, Fujii Masao  wrote:
> On Mon, Oct 31, 2011 at 5:23 PM, Simon Riggs  wrote:
>> On Mon, Oct 31, 2011 at 7:38 AM, Fujii Masao  wrote:
>>
 In 9.2 the presence of recovery.conf can and therefore should continue
 to act as it does in 9.1.
>>>
>>> This means that recovery.conf is renamed to recovery.done at the end of
>>> recovery. IOW, all settings in recovery.conf are reset when recovery ends.
>>> Then if you run "pg_ctl reload" after recovery, you'll get something like
>>> the following error message and the reload will always fail;
>>>
>>>   LOG:  parameter "standby_mode" cannot be changed without restarting
>>> the server
>>
>>> To resolve this issue,
>>
>> This issue exists whether or not we have recovery.conf etc., so yes,
>> we must solve the problem.
>
> No. If we don't have recovery.conf, all parameters exist in postgresql.conf.
> The above issue would not occur unless a user makes a mistake, e.g.,
> change the value of the parameter which cannot be changed without
> the server restart.

I don't understand what you are saying. You seem to be suggesting that
it would be OK for someone to set standby_mode = on and reload the
config and for that to go completely unremarked. If we are adding new
parameters to postgresql.conf then its clear that some people will
misconfigure them and we need to have error messages for that.

If you change a parameter that only has effect during recovery then
must get an error if it is changed during normal running.

That is the reason we need to mark the recovery parameters with a new
flag, so that we can ignore any errors caused by changing them.

That has nothing at all to do with recovery.conf - you need it even if
you put everything in postgresql.conf.

>>> I think that we need to introduce new GUC flag
>>> indicating parameters are used only during recovery, and need to define
>>> recovery parameters with that flag. Whenever "pg_ctl reload" is executed,
>>> all the processes check whether recovery is in progress, and ignore
>>> silently the parameters with that flag if not. I'm not sure how easy we
>>> can implement this. In addition to introducing that flag, we might need to
>>> change some processes which cannot access to the shared memory so that
>>> they can. Otherwise, they cannot know whether recovery is in progress.
>>> Or we might need to change them so that they always ignore recovery
>>> parameters.
>>
>> The postmaster knows whether its in recovery or not without checking
>> shared memory. Various postmaster states describe this. If not
>> postmaster, other backends can run recoveryinprogress() as normal.
>
> AFAIR archiver and syslogger cannot access to the shared memory, i.e.,
> they cannot run RecoveryInProgress(). They don't use any recovery
> parameters for now, so we can change them so that they always ignore
> those parameters. Though I'm not inclined to add the process-specific code
> like the following into the GUC mechanism as much as possible.
>
>    if (I am postmaster)
>    {
>        if (recovery is NOT in progress)
>            reset the recovery parameters;
>    }
>    else if (I am archiver or syslogger)
>        /* always ignore */
>    else
>    {
>        if (recovery is NOT in progress)
>            reset the recovery parameters;
>    }
>

I don't see the problem. The code above could easily be simplified and
only needs to exist in one place, not copied around for each GUC.

When we had recovery.conf and postgresql.conf you knew which
parameters took effects at specific times. That metadata needs to be
added back into the system so we can report errors properly.

Considering the amount of code we will be removing, a couple of extra
lines seems trivial.

AFAICS we need to reset all recovery parameters at the end of recovery
anyway. Having SHOW recovery_target_xid return a value other than 0 is
not appropriate during normal running, same for standby_mode and the
other parameters.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-31 Thread Fujii Masao

On Mon, Oct 31, 2011 at 5:23 PM, Simon Riggs  wrote:
> On Mon, Oct 31, 2011 at 7:38 AM, Fujii Masao  wrote:
>
>>> In 9.2 the presence of recovery.conf can and therefore should continue
>>> to act as it does in 9.1.
>>
>> This means that recovery.conf is renamed to recovery.done at the end of
>> recovery. IOW, all settings in recovery.conf are reset when recovery ends.
>> Then if you run "pg_ctl reload" after recovery, you'll get something like
>> the following error message and the reload will always fail;
>>
>>   LOG:  parameter "standby_mode" cannot be changed without restarting
>> the server
>
>> To resolve this issue,
>
> This issue exists whether or not we have recovery.conf etc., so yes,
> we must solve the problem.

No. If we don't have recovery.conf, all parameters exist in postgresql.conf.
The above issue would not occur unless a user makes a mistake, e.g.,
change the value of the parameter which cannot be changed without
the server restart.

>> I think that we need to introduce new GUC flag
>> indicating parameters are used only during recovery, and need to define
>> recovery parameters with that flag. Whenever "pg_ctl reload" is executed,
>> all the processes check whether recovery is in progress, and ignore
>> silently the parameters with that flag if not. I'm not sure how easy we
>> can implement this. In addition to introducing that flag, we might need to
>> change some processes which cannot access to the shared memory so that
>> they can. Otherwise, they cannot know whether recovery is in progress.
>> Or we might need to change them so that they always ignore recovery
>> parameters.
>
> The postmaster knows whether its in recovery or not without checking
> shared memory. Various postmaster states describe this. If not
> postmaster, other backends can run recoveryinprogress() as normal.

AFAIR archiver and syslogger cannot access to the shared memory, i.e.,
they cannot run RecoveryInProgress(). They don't use any recovery
parameters for now, so we can change them so that they always ignore
those parameters. Though I'm not inclined to add the process-specific code
like the following into the GUC mechanism as much as possible.

if (I am postmaster)
{
if (recovery is NOT in progress)
reset the recovery parameters;
}
else if (I am archiver or syslogger)
/* always ignore */
else
{
if (recovery is NOT in progress)
reset the recovery parameters;
}

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-31 Thread Robert Treat

On Mon, Oct 31, 2011 at 3:19 PM, Simon Riggs  wrote:
> On Mon, Oct 31, 2011 at 7:05 PM, Josh Berkus  wrote:
>
>> If it's possible to run a replica without having a recovery.conf file,
>> then I'm fine with your solution.  If it's not, then I find your
>> solution not to be a solution at all.
>
> Then you are fine with the solution - not mine alone, just the sum of
> everybody's inputs.
>
> So we can teach the new way, while supporting the old way a while longer.
>

In most cases we either break backwards compatibility or require some
type of switch to turn on backwards compatibility for those who want
it. While the above plan tries to do one better, it leaves me feeling
that the thing I don't like about this is that it sounds like you are
forcing backwards compatibility on people who would much rather just
do things the new way. Given that, I foresee a whole new generation of
confused users who end up setting their configs one way only to have
someone else set the same config in the other file, or some tool dump
out some config file, overriding what was really intended. This will
also make things *harder* for those tool providers you are trying to
help, as they will be forced to support the behavior *both ways*. I'd
much rather see some type of switch which turns on the old behavior
for those who really want it, because while you can teach the new
behavior, if you can't prevent the old behavior, you're creating
operational headaches for yourself.

Robert Treat
conjecture: xzilla.net
consulting: omniti.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-31 Thread Simon Riggs

On Mon, Oct 31, 2011 at 7:05 PM, Josh Berkus  wrote:

> If it's possible to run a replica without having a recovery.conf file,
> then I'm fine with your solution.  If it's not, then I find your
> solution not to be a solution at all.

Then you are fine with the solution - not mine alone, just the sum of
everybody's inputs.

So we can teach the new way, while supporting the old way a while longer.

> Sure.  recovery.conf worked fine for PITR.  We've just overextended it
> for other purposes.

Agreed.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-31 Thread Josh Berkus

Simon,

> Everybody agrees a neater way of invoking standby mode would be good.

I don't think this goes far enough.  The whole
recovery.conf/recovery.done thing is a serious problem for automated
management of servers and automated failover.  So it's not just "a
neater way would be good" but "using recovery.conf as a trigger file is
a broken idea and needs to be changed."

> These things are announced as deprecated and will be removed when we
> go to release 10.0
> * trigger_file
> * standby_mode
> * recovery.conf indicates standby

So you're idea is that people who don't want recovery.conf to be used as
a trigger file would not have the file at all, but would have something
like "replication.conf" instead?

If it's possible to run a replica without having a recovery.conf file,
then I'm fine with your solution.  If it's not, then I find your
solution not to be a solution at all.

> recovery.conf should continue to be required to perform a PITR. If we
> place the recovery_target parameters into postgresql.conf we will have
> no way to differentiate between (1) a recovery that has successfully
> completed then crashed and (2) a user-specified recovery, which was
> the original rationale for its use. This is OK, since we now encourage
> people to enter a recovery by creating recovery.conf and for entering
> a standby to use a new cleaner API without the confusing use of the
> word "recovery".

Sure.  recovery.conf worked fine for PITR.  We've just overextended it
for other purposes.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-31 Thread Simon Riggs

On Mon, Oct 31, 2011 at 7:38 AM, Fujii Masao  wrote:

>> In 9.2 the presence of recovery.conf can and therefore should continue
>> to act as it does in 9.1.
>
> This means that recovery.conf is renamed to recovery.done at the end of
> recovery. IOW, all settings in recovery.conf are reset when recovery ends.
> Then if you run "pg_ctl reload" after recovery, you'll get something like
> the following error message and the reload will always fail;
>
>   LOG:  parameter "standby_mode" cannot be changed without restarting
> the server

> To resolve this issue,

This issue exists whether or not we have recovery.conf etc., so yes,
we must solve the problem.


> I think that we need to introduce new GUC flag
> indicating parameters are used only during recovery, and need to define
> recovery parameters with that flag. Whenever "pg_ctl reload" is executed,
> all the processes check whether recovery is in progress, and ignore
> silently the parameters with that flag if not. I'm not sure how easy we
> can implement this. In addition to introducing that flag, we might need to
> change some processes which cannot access to the shared memory so that
> they can. Otherwise, they cannot know whether recovery is in progress.
> Or we might need to change them so that they always ignore recovery
> parameters.

The postmaster knows whether its in recovery or not without checking
shared memory. Various postmaster states describe this. If not
postmaster, other backends can run recoveryinprogress() as normal.

It makes sense to have a new flag and that is easily created and used.


> Another simple but somewhat restricted approach is to read and set
> all parameters specified in recovery.conf by using PGC_S_OVERRIDE.
> If we do this, those parameters cannot be changed after startup
> even if recovery.conf is renamed. But the problem is that a user also
> cannot change their settings by reloading the configuration files. This is
> obviously a restriction. But it doesn't break any backward compatibility,
> I believe. No? If a user prefers new functionality (i.e., reload recovery
> parameters) rather than the backward compatibility, he/she can specify
> parameters in postgresql.conf. Thought?

No need to create problems.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-31 Thread Fujii Masao

On Sat, Oct 29, 2011 at 7:54 PM, Simon Riggs  wrote:
> On Fri, Sep 9, 2011 at 10:56 AM, Fujii Masao  wrote:
>
>> In previous discussion, we've reached the consensus that we should unite
>> recovery.conf and postgresql.conf. The attached patch does that. The
>> patch is WIP, I'll have to update the document, but if you notice something,
>> please feel free to comment.
>
> My short summary of the thread is

Thanks!

> In 9.1 we added "pg_ctl promote" as a better way of indicating
> failover/switchover. When we did that we kept the trigger_file
> parameter added in 9.0, which shows it is possible to add a new API
> without breaking backwards compatibility.
>
> We should add a "pg_ctl standby" command as a better way of indicating
> starting up (also described as triggering) standby mode. We keep
> standby_mode parameter.  There is no difference here between file
> based and stream based replication: you can have file, stream or both
> file and stream (as intended).
> In this mode the recovery target parameters are *ignored* even if
> specified (explained below).
> http://developer.postgresql.org/pgdocs/postgres/recovery-target-settings.html

Agreed to add "pg_ctl standby". I think that this can be committed
separately from the change of recovery.conf.

> In 9.2 the presence of recovery.conf can and therefore should continue
> to act as it does in 9.1.

This means that recovery.conf is renamed to recovery.done at the end of
recovery. IOW, all settings in recovery.conf are reset when recovery ends.
Then if you run "pg_ctl reload" after recovery, you'll get something like
the following error message and the reload will always fail;

   LOG:  parameter "standby_mode" cannot be changed without restarting
the server

To resolve this issue, I think that we need to introduce new GUC flag
indicating parameters are used only during recovery, and need to define
recovery parameters with that flag. Whenever "pg_ctl reload" is executed,
all the processes check whether recovery is in progress, and ignore
silently the parameters with that flag if not. I'm not sure how easy we
can implement this. In addition to introducing that flag, we might need to
change some processes which cannot access to the shared memory so that
they can. Otherwise, they cannot know whether recovery is in progress.
Or we might need to change them so that they always ignore recovery
parameters.

Another simple but somewhat restricted approach is to read and set
all parameters specified in recovery.conf by using PGC_S_OVERRIDE.
If we do this, those parameters cannot be changed after startup
even if recovery.conf is renamed. But the problem is that a user also
cannot change their settings by reloading the configuration files. This is
obviously a restriction. But it doesn't break any backward compatibility,
I believe. No? If a user prefers new functionality (i.e., reload recovery
parameters) rather than the backward compatibility, he/she can specify
parameters in postgresql.conf. Thought?

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-29 Thread Simon Riggs

On Fri, Sep 9, 2011 at 10:56 AM, Fujii Masao  wrote:

> In previous discussion, we've reached the consensus that we should unite
> recovery.conf and postgresql.conf. The attached patch does that. The
> patch is WIP, I'll have to update the document, but if you notice something,
> please feel free to comment.

My short summary of the thread is

Fujii proposes we allow parameters currently in recovery.conf to be
specified in postgresql.conf. All agree to that. Fujii suggests that
if we have both postgresql.conf and recovery.conf then recovery.conf
should contain overrides.

Fujii then suggests that if such an override exists, then SHOW would
not work properly. Magnus is rightly horrified and many speak against
allowing recovery.conf to continue to exist for this reason.  I note
that if recovery.conf is an include file of postgresql.conf then the
overrides would work correctly, just as if postgresql.conf had
multiple settings for that parameter. So the premise is incorrect, so
the conclusion is not relevant.

Simon, JD, Greg Stark speak in favour of the usefulness of having a
recovery.conf separate from postgresql.conf. Tatsuo confirms pgpool
uses this.

Simon, Fujii, Peter agree an automatic include of recovery.conf would be useful

Robert points out that pg_ctl promote was a good feature

Simon, JD say that backwards compatibility is important

Everybody agrees a neater way of invoking standby mode would be good.

Peter points out that including recovery target parameters in
postgresql.conf would be difficult and require manual editing, and
also that pg_ctl -o is not a suitable interface.

The thread also includes a variety of other alternate ideas,
misunderstandings and other commentary.

- - -

My thoughts on how to resolve this are...

Everybody agrees these two points:

* Allow recovery parameters to be handled same way as other GUCs, and
specified in postgresql.conf if desired.

* Allow parameters to be reloaded at SIGHUP and visible using SHOW.

Those two things do not themselves force us to break backwards compatibility.

We also agree that we want a neater way to startup in standby mode.

In 9.1 we added "pg_ctl promote" as a better way of indicating
failover/switchover. When we did that we kept the trigger_file
parameter added in 9.0, which shows it is possible to add a new API
without breaking backwards compatibility.

We should add a "pg_ctl standby" command as a better way of indicating
starting up (also described as triggering) standby mode. We keep
standby_mode parameter.  There is no difference here between file
based and stream based replication: you can have file, stream or both
file and stream (as intended).
In this mode the recovery target parameters are *ignored* even if
specified (explained below).
http://developer.postgresql.org/pgdocs/postgres/recovery-target-settings.html

In 9.2 the presence of recovery.conf can and therefore should continue
to act as it does in 9.1. This should be automatically included at the
end of postgresql.conf, which naturally and with no additional code
allows us to override settings, with overrides visible by SHOW. We
don't make any specific checks to see if someone has added a
postgresql.conf parameter in there. If there is a recovery target
parameter in recovery.conf we enter recovery, otherwise we operate as
a standby. recovery.conf is no longer *required* for standby modes.

These things are announced as deprecated and will be removed when we
go to release 10.0
* trigger_file
* standby_mode
* recovery.conf indicates standby

recovery.conf should continue to be required to perform a PITR. If we
place the recovery_target parameters into postgresql.conf we will have
no way to differentiate between (1) a recovery that has successfully
completed then crashed and (2) a user-specified recovery, which was
the original rationale for its use. This is OK, since we now encourage
people to enter a recovery by creating recovery.conf and for entering
a standby to use a new cleaner API without the confusing use of the
word "recovery".

I think that meets all requirements, as far as technically possible.

Best Regards

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-12 Thread Josh Berkus

Simon,

I haven't see a response from you on a proposed way to keep backwards
compatibility with recovery.conf as a trigger file, while also
eliminating its trigger status as an unmanagable misfeature.  As far as
I can tell, that's the one area where we *cannot* maintain backwards
compatibility.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-11 Thread Bruce Momjian

Simon Riggs wrote:
> On Tue, Oct 11, 2011 at 9:28 PM, Bruce Momjian  wrote:
> 
> > Standard conforming strings
> > was tricky because it was more user-facing, or certainly SQL-facing.
> 
> Why is SQL more important than backup?

Because the percentage of database users it affects is different. 
Administrators know when they are installing a new version of Postgres
and already are probably changing these configuration files. 
Application binaries and perhaps application developers are not as aware
of a change, and there are a far higher percentage of them in an
organization than administrators.

> There is no good reason to do this so quickly.

I just gave you a reason above, and as I said, doing backward
compatibility can make the system more complex.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-11 Thread Simon Riggs

On Tue, Oct 11, 2011 at 9:28 PM, Bruce Momjian  wrote:

> Standard conforming strings
> was tricky because it was more user-facing, or certainly SQL-facing.

Why is SQL more important than backup?

There is no good reason to do this so quickly.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-11 Thread Bruce Momjian

Simon Riggs wrote:
> On Tue, Oct 11, 2011 at 3:29 PM, Bruce Momjian  wrote:
> 
> > As much as I appreciate Simon's work in this area, I think we are still
> > unclear if keeping backward-compatibility is worth the complexity
> > required for future users. ?Historically we have been bold in changing
> > postgresql.conf settings to improve clarity, and that approach has
> > served us well.
> 
> You raise a good point. First, thank you for the respectful comment;
> my viewpoint is not formed from resistance to change per se, even if
> may appear to be so.  Thank you for raising that possibility to allow
> me to explain and refute that.
> 
> I am genuinely concerned that we show respect to downstream software
> that uses our APIs and have no personal or corporate ulterior motive.
> 
> Most people are used to the 3 year cycle of development on which
> SQLServer and Oracle have now standardised. Our 1 year cycle provides
> a considerable benefit in agility, but it also provides for x3
> complexity in release management and a continual temptation to change
> for no good reason. I want to encourage people to adopt our APIs, not
> give them a headache for attempting to do so. We know that software
> exists that follows the previous API and we should take steps to
> deprecate that across multiple releases, with appropriate notice, just
> as we do in other cases, such as standard conforming strings where our
> lack of boldness is appropriate.

Well, let me be specific.  Around 2003 to 2006, we added many new
configuration parameters for logging, which required renaming or
removing older parameters.  There really wasn't a smooth way to allow
for this to be done without impacting users, and the current system we
have enjoyed since 2006 is logical only because we made the changes
necessary.

We can look at trying to phase changes in, but often the phasing becomes
more complicated that just doing the change.  Logging parameter changes
were easier because it was assumed logging was an admin-only task, as I
assume pitr and replication are as well.  Standard conforming strings
was tricky because it was more user-facing, or certainly SQL-facing.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-11 Thread Simon Riggs

On Tue, Oct 11, 2011 at 3:29 PM, Bruce Momjian  wrote:

> As much as I appreciate Simon's work in this area, I think we are still
> unclear if keeping backward-compatibility is worth the complexity
> required for future users.  Historically we have been bold in changing
> postgresql.conf settings to improve clarity, and that approach has
> served us well.

You raise a good point. First, thank you for the respectful comment;
my viewpoint is not formed from resistance to change per se, even if
may appear to be so.  Thank you for raising that possibility to allow
me to explain and refute that.

I am genuinely concerned that we show respect to downstream software
that uses our APIs and have no personal or corporate ulterior motive.

Most people are used to the 3 year cycle of development on which
SQLServer and Oracle have now standardised. Our 1 year cycle provides
a considerable benefit in agility, but it also provides for x3
complexity in release management and a continual temptation to change
for no good reason. I want to encourage people to adopt our APIs, not
give them a headache for attempting to do so. We know that software
exists that follows the previous API and we should take steps to
deprecate that across multiple releases, with appropriate notice, just
as we do in other cases, such as standard conforming strings where our
lack of boldness is appropriate.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-11 Thread Josh Berkus

On 10/10/11 9:53 PM, Fujii Masao wrote:
> Or you think that, to keep the backward compatibility completely,
> recovery.conf should be used as not only a configuration file but also a
> recovery trigger one and it should be renamed to recovery.done at
> the end of recovery?

That's precisely my point.  The trigger file nature of recovery.conf is
a problem in itself, and I don't see any way to support that and fix it
at the same time.  Maybe Simon can?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-11 Thread Bruce Momjian

Fujii Masao wrote:
> On Tue, Oct 11, 2011 at 6:37 AM, Simon Riggs  wrote:
> > On Mon, Oct 10, 2011 at 6:52 PM, Josh Berkus  wrote:
> >
> >>> Tatsuo/Josh/Robert also discussed how recovery.conf can be used to
> >>> provide parameters solely for recovery. That is difficult to do
> >>> without causing all downstream tools to make major changes in the ways
> >>> they supply parameters.
> >>
> >> Actually, this case is easily solved by an "include recovery.conf"
> >> parameter. ?So it's a non-issue.
> >
> > That is what I've suggested and yes, doing that is straightforward.
> 
> Even if we do that, you still need to modify the tool so that it can handle
> the recovery trigger file. recovery.conf is used as just a configuration file
> (not recovery trigger file at all). It's not renamed to recovery.done at the
> end of recovery. If the tool depends on the renaming from recovery.conf
> to recovery.done, it also would need to be modified. If the tool needs to
> be changed anyway, why do you hesitate in changing it so that it adds
> "include recovery.conf" into postgresql.conf automatically?
> 
> Or you think that, to keep the backward compatibility completely,
> recovery.conf should be used as not only a configuration file but also a
> recovery trigger one and it should be renamed to recovery.done at
> the end of recovery?

As much as I appreciate Simon's work in this area, I think we are still
unclear if keeping backward-compatibility is worth the complexity
required for future users.  Historically we have been bold in changing
postgresql.conf settings to improve clarity, and that approach has
served us well.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-10 Thread Fujii Masao

On Tue, Oct 11, 2011 at 6:37 AM, Simon Riggs  wrote:
> On Mon, Oct 10, 2011 at 6:52 PM, Josh Berkus  wrote:
>
>>> Tatsuo/Josh/Robert also discussed how recovery.conf can be used to
>>> provide parameters solely for recovery. That is difficult to do
>>> without causing all downstream tools to make major changes in the ways
>>> they supply parameters.
>>
>> Actually, this case is easily solved by an "include recovery.conf"
>> parameter.  So it's a non-issue.
>
> That is what I've suggested and yes, doing that is straightforward.

Even if we do that, you still need to modify the tool so that it can handle
the recovery trigger file. recovery.conf is used as just a configuration file
(not recovery trigger file at all). It's not renamed to recovery.done at the
end of recovery. If the tool depends on the renaming from recovery.conf
to recovery.done, it also would need to be modified. If the tool needs to
be changed anyway, why do you hesitate in changing it so that it adds
"include recovery.conf" into postgresql.conf automatically?

Or you think that, to keep the backward compatibility completely,
recovery.conf should be used as not only a configuration file but also a
recovery trigger one and it should be renamed to recovery.done at
the end of recovery?

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-10 Thread Simon Riggs

On Mon, Oct 10, 2011 at 6:52 PM, Josh Berkus  wrote:

>> Tatsuo/Josh/Robert also discussed how recovery.conf can be used to
>> provide parameters solely for recovery. That is difficult to do
>> without causing all downstream tools to make major changes in the ways
>> they supply parameters.
>
> Actually, this case is easily solved by an "include recovery.conf"
> parameter.  So it's a non-issue.

That is what I've suggested and yes, doing that is straightforward.

If you mean "do that in a program" if we had a problem with adding
parameters, we also have a problem adding an include.

We should avoid breaking programs which we have no reason to break.
Stability is good, change without purpose is not.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-10 Thread Josh Berkus

On 10/10/11 10:52 AM, Josh Berkus wrote:
> So after debugging some of our failover scripts, here's the real-world
> problems I'm trying to solve.  These design flaws are issues which cause
> automated failover or failback to abort, leading to unexpected downtime,
> so they are not just issues of neatness:

That's "automated failover or *manual* failback".  I never, ever
recommend automated failback.  Just FYI.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-10 Thread Josh Berkus

Simon,

> Tatsuo/Josh/Robert also discussed how recovery.conf can be used to
> provide parameters solely for recovery. That is difficult to do
> without causing all downstream tools to make major changes in the ways
> they supply parameters.

Actually, this case is easily solved by an "include recovery.conf"
parameter.  So it's a non-issue.

> Keeping our APIs relatively stable is important to downstream tools. I
> have no objection to a brave new world, as long as you don't chuck out
> the one that works right now. Breaking APIs needs a good reason and
> I've not seen one discussed anywhere. No problem with immediately
> deprecating the old API and declare is planned to be removed in
> release 10.0.

So after debugging some of our failover scripts, here's the real-world
problems I'm trying to solve.  These design flaws are issues which cause
automated failover or failback to abort, leading to unexpected downtime,
so they are not just issues of neatness:

1. Recovery.conf being both a configuration file AND a trigger to
initiate recovery mode, preventing us from separating configuration
management from failover.

2. The inability to read recovery.conf parameters via SQL on a hot
standby, forcing us to parse the file to find out its settings, or guess.

(1) is a quite serious issue; it effectively makes recovery.conf
impossible to integrate with puppet and other configuration management
frameworks.  I also don't see a way to fix it without breaking backwards
compatibility.

BTW, I'm not criticizing the original design for this.  We simply didn't
know better until lots of people were using these tools in production.
But it's time to fix them, and the longer we wait, the more painful it
will be.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-09 Thread Simon Riggs

On Tue, Sep 27, 2011 at 10:34 AM, Fujii Masao  wrote:
> On Mon, Sep 26, 2011 at 7:45 PM, Peter Eisentraut  wrote:
>> On sön, 2011-09-25 at 12:58 -0400, Tom Lane wrote:
>>> And it's not like we don't break configuration file
>>> contents in most releases anyway, so I really fail to see why this one
>>> has suddenly become sacrosanct.
>>
>> Well, there is a slight difference.  Changes in postgresql.conf
>> parameter names and settings are adjusted automatically for me by my
>> package upgrade script.  If we, say, changed the names of recovery.conf
>> parameters, I'd have to get a new version of my $SuperReplicationTool.
>> That tool could presumably look at PG_VERSION and put a recovery.conf
>> with the right spellings in the right place.
>>
>> But if we completely change the way the replication configuration
>> interacts, it's not clear that a smooth upgrade is possible without
>> significant effort.  That said, I don't see why it wouldn't be possible,
>> but let's design with upgradability in mind, instead of claiming that we
>> have never supported upgrades of this kind anyway.
>
> Currently recovery.conf has two roles:
>
> #1. recovery.conf is used as a trigger file to enable archive recovery.
>      At the end of recovery, recovery.conf is renamed to recovery.done.
>
> #2. recovery.conf is used as a configuration file for recovery parameters.
>
> Which role do you think we should support in 9.2 because of the backward
> compatibility? Both? Unless I misunderstand the discussion so far, Tom and
> Robert (and I) agree to get rid of both. Simon seems to agree to remove
> only the former role, but not the latter. How about you? If you agree to
> remove the former, too, let's focus on the discussion about whether the
> latter role should be supported in 9.2.

Tatsuo/Josh/Robert also discussed how recovery.conf can be used to
provide parameters solely for recovery. That is difficult to do
without causing all downstream tools to make major changes in the ways
they supply parameters.

Keeping our APIs relatively stable is important to downstream tools. I
have no objection to a brave new world, as long as you don't chuck out
the one that works right now. Breaking APIs needs a good reason and
I've not seen one discussed anywhere. No problem with immediately
deprecating the old API and declare is planned to be removed in
release 10.0.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-10-08 Thread Jeff Janes

On Tue, Sep 27, 2011 at 1:33 AM, Fujii Masao  wrote:
>
> Though there is still ongonig discussion, since there is no objection about
> the above two changes, I revised the patch that way. And I fixed the minor
> bug handling the default value of recovery_target_timeline wrongly.
> Attached is the revised version of the patch.

This patch no longer applies as it conflicts with the following commit:

commit d56b3afc0376afe491065d9eca6440b3cc7b1346
Author: Tom Lane 
Date:   Sun Oct 2 16:50:04 2011 -0400

Restructure error handling in reading of postgresql.conf.

Cheers,

Jeff

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-27 Thread Fujii Masao

On Mon, Sep 26, 2011 at 7:45 PM, Peter Eisentraut  wrote:
> On sön, 2011-09-25 at 12:58 -0400, Tom Lane wrote:
>> And it's not like we don't break configuration file
>> contents in most releases anyway, so I really fail to see why this one
>> has suddenly become sacrosanct.
>
> Well, there is a slight difference.  Changes in postgresql.conf
> parameter names and settings are adjusted automatically for me by my
> package upgrade script.  If we, say, changed the names of recovery.conf
> parameters, I'd have to get a new version of my $SuperReplicationTool.
> That tool could presumably look at PG_VERSION and put a recovery.conf
> with the right spellings in the right place.
>
> But if we completely change the way the replication configuration
> interacts, it's not clear that a smooth upgrade is possible without
> significant effort.  That said, I don't see why it wouldn't be possible,
> but let's design with upgradability in mind, instead of claiming that we
> have never supported upgrades of this kind anyway.

Currently recovery.conf has two roles:

#1. recovery.conf is used as a trigger file to enable archive recovery.
  At the end of recovery, recovery.conf is renamed to recovery.done.

#2. recovery.conf is used as a configuration file for recovery parameters.

Which role do you think we should support in 9.2 because of the backward
compatibility? Both? Unless I misunderstand the discussion so far, Tom and
Robert (and I) agree to get rid of both. Simon seems to agree to remove
only the former role, but not the latter. How about you? If you agree to
remove the former, too, let's focus on the discussion about whether the
latter role should be supported in 9.2.

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-26 Thread Andrew Dunstan




On 09/25/2011 02:39 PM, Joshua Berkus wrote:

There might be a use case for a separate directive include_if_exists,
or some such name.  But I think the user should have to tell us very
clearly that it's okay for the file to not be found.

Better to go back to include_directory, then.





I rather like Tom's suggestion of include_if_exists.

There might be a case for include_directory, but I think that needs to 
be made separately.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-26 Thread Peter Eisentraut

On sön, 2011-09-25 at 12:58 -0400, Tom Lane wrote:
> And it's not like we don't break configuration file
> contents in most releases anyway, so I really fail to see why this one
> has suddenly become sacrosanct. 

Well, there is a slight difference.  Changes in postgresql.conf
parameter names and settings are adjusted automatically for me by my
package upgrade script.  If we, say, changed the names of recovery.conf
parameters, I'd have to get a new version of my $SuperReplicationTool.
That tool could presumably look at PG_VERSION and put a recovery.conf
with the right spellings in the right place.

But if we completely change the way the replication configuration
interacts, it's not clear that a smooth upgrade is possible without
significant effort.  That said, I don't see why it wouldn't be possible,
but let's design with upgradability in mind, instead of claiming that we
have never supported upgrades of this kind anyway.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-26 Thread Peter Eisentraut

On lör, 2011-09-24 at 14:02 +0100, Simon Riggs wrote:
> The semantics are clear: recovery.conf is read first, then
> postgresql.conf. It's easy to implement (1 line of code) and easy to
> understand.

What's clear about that?  My intuition would have been that
recovery.conf is read second.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-26 Thread Peter Eisentraut

On lör, 2011-09-24 at 13:04 -0400, Tom Lane wrote:
> > What if we modified pg_ctl to allow passing configuration parameters
> > through to postmaster,
> 
> You mean like pg_ctl -o?

Note that pg_ctl -o saves the parameters it uses and reapplies them
after a restart.  So this is not really the way to apply parameter
settings only once for recovery.  (Or at least it has the potential to
be a very confusing way.)

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

Joshua Berkus  writes:
> include_if_exists certainly solves the recovery.conf/recovery.done problem.  
> We can even phase it out, like this:

> 9.2: include_if_exists = 'recovery.conf' in the default postgresql.conf file.
> 9.3: include_if_exists = 'recovery.conf' commented out by default
> 9.4: renaming recovery.conf to recovery.done by core PG code removed.

> This gives users/vendors 3 years to update their scripts to remove dependence 
> on recovery.conf.  I'm afraid that I agree with Simon that there's already a 
> whole buncha 3rd-party code out there to support the current system.

If there is indeed code out there that depends on the current system,
why do you think that waiting several releases to change it will make
things better?  I think it's more likely that we'd just have *more* code
that needs to be changed, and no reduction in the pain level when the
transition does finally happen.

In any case, I thought we'd agreed that the use of that file as a flag
should go away now, so I quite fail to understand your comment about 9.4.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-25 Thread Joshua Berkus



> I rather like Tom's suggestion of include_if_exists.

include_if_exists certainly solves the recovery.conf/recovery.done problem.  We 
can even phase it out, like this:

9.2: include_if_exists = 'recovery.conf' in the default postgresql.conf file.
9.3: include_if_exists = 'recovery.conf' commented out by default
9.4: renaming recovery.conf to recovery.done by core PG code removed.

This gives users/vendors 3 years to update their scripts to remove dependence 
on recovery.conf.  I'm afraid that I agree with Simon that there's already a 
whole buncha 3rd-party code out there to support the current system.

--Josh Berkus 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-25 Thread Joshua Berkus



> There might be a use case for a separate directive include_if_exists,
> or some such name.  But I think the user should have to tell us very
> clearly that it's okay for the file to not be found.

Better to go back to include_directory, then.

--Josh Berkus

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

Joshua Berkus  writes:
> What happens currently if we have an \include in postgresql.conf for a file 
> which doesn't exist?  Is it ignored, or do we error out?

It's an error, and I think changing that would be a really bad idea.

> If it could just be ignored, maybe with a note in the logs, then we could be 
> a lot more flexible.

There might be a use case for a separate directive include_if_exists,
or some such name.  But I think the user should have to tell us very
clearly that it's okay for the file to not be found.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

2011-09-25 Thread Joshua Berkus

Folks,

What happens currently if we have an \include in postgresql.conf for a file 
which doesn't exist?  Is it ignored, or do we error out?

If it could just be ignored, maybe with a note in the logs, then we could be a 
lot more flexible.

--Josh Berkus

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf

Robert Haas  writes:
> On Sep 24, 2011, at 1:04 PM, Tom Lane  wrote:
>> I don't exactly buy this argument.  If postgresql.conf is hard to
>> machine-edit, why is recovery.conf any easier?

> Because you generally just write a brand-new file, without worrying
> about preserving existing settings. You aren't really editing at all,
> just writing.

If that's all the requirement is, it's trivial to implement.

1. Write your-random-configuration-settings into recovery.conf (or any
other file name you choose ... something named after your tool would be
a better idea).

2. Temporarily append "include recovery.conf" to the end of
postgresql.conf.  Restart server.

3. When done, remove "include recovery.conf" from the end of
postgresql.conf.

The hard cases involve merging user-supplied and tool-supplied settings,
but "let's overwrite recovery.conf in toto" never would have been able
to handle such cases either.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unite recovery.conf and postgresql.conf