Re: [HACKERS] pg_rewind proposed scope and interface changes

2017-09-15 Thread Chris Travers
On Wed, Sep 13, 2017 at 6:28 AM, Michael Paquier 
wrote:

> On Tue, Sep 12, 2017 at 11:52 PM, Chris Travers
>  wrote:
> > Additionally the wal, xact, timestamp and logical directories must be
> > processed in some way.
>
> To what does the term "logical directories" refer to?
>
> >   * if --wal=sync the directories are processed the way they are today
> >   * if --wal=clear then the contents of the directories are cleared and
> > replication is assumed to be used to bring the system up after.  Note
> this
> > will need to come with warning about the need for replication slots.
>
> Hm. I am not sure in what --wal=clear is helpful. Keeping around WAL
> segments from the point of the last checkpoint where WAL forked up to
> the point where WAL has forked is helpful, because you don't need to
> copy again those WAL segments, be they come from an archive or from
> streaming. Copying a set of WAL segments during the rewind of the new
> timeline is helpful as well because you don't need to do the copy
> again. One configuration where this is helpful is that there is
> already an archive local to the target server available with the
> segments of the new timeline available.
>

so maybe clear should just clear diverged wal files.  That makes some
sense.

>
> > Base, global, pg_tablespace
> >
> > With
> > pg_wal, pg_xact, pg_commit_ts, pg_logical added if wal strategy is set to
> > sync.
>
> Skipping some directories in a way similar to what a base backup does
> would be nicer I think. We already have a list of those in
> basebackup.c in the shape of excludeDirContents and excludeFiles. I
> think that it would be a good idea to export those into a header that
> pg_rewind could include, and refer to in order to exclude them when
> fetching a set of files. At the end of the day, a rewind is a kind of
> base backup in itself, and this patch would already serve well a lot
> of people.
>

I think there are several reasons not to just do this based on base
backup.  For base backup there may be some things we restore as such that
we don't want to restore with pg_rewind.  Configuration files, logs, and
replication slots are the ones that immediately come to mind.  Given that
configuration files can have arbitrary names and locations, expecting third
party failover tooling like repmgr or patroni to pick up on such policies
strikes me as asking a bit much.  I think we are better off with a program
that solves one problem well and solves it right and the problem is
rewinding a system so that it can follow a replica that was previously
following it.

>
> Having on top of that a way to exclude a wanted set of files and the
> log directory (for example: should we look at log_directory and
> exclude it from the fetched paths if it is not an absolute path?),
> which is smart enough to take care of not removing paths critical for
> a rewind like anything in base/, then you are good to go with a
> full-blown tool that I think would serve the purposes you are looking
> for.
>

If you are going to go this route, I think you need to look at the config
files themselves, parse them, and possibly look at the command line by
which PostgreSQL is started.Otherwise I don't know how you expect this
to be configured


> --
> Michael
>



-- 
Best Regards,
Chris Travers
Database Administrator

Tel: +49 162 9037 210 | Skype: einhverfr | www.adjust.com
Saarbrücker Straße 37a, 10405 Berlin


Re: [HACKERS] pg_rewind proposed scope and interface changes

2017-09-12 Thread Michael Paquier
On Tue, Sep 12, 2017 at 11:52 PM, Chris Travers
 wrote:
> Additionally the wal, xact, timestamp and logical directories must be
> processed in some way.

To what does the term "logical directories" refer to?

>   * if --wal=sync the directories are processed the way they are today
>   * if --wal=clear then the contents of the directories are cleared and
> replication is assumed to be used to bring the system up after.  Note this
> will need to come with warning about the need for replication slots.

Hm. I am not sure in what --wal=clear is helpful. Keeping around WAL
segments from the point of the last checkpoint where WAL forked up to
the point where WAL has forked is helpful, because you don't need to
copy again those WAL segments, be they come from an archive or from
streaming. Copying a set of WAL segments during the rewind of the new
timeline is helpful as well because you don't need to do the copy
again. One configuration where this is helpful is that there is
already an archive local to the target server available with the
segments of the new timeline available.

> Base, global, pg_tablespace
>
> With
> pg_wal, pg_xact, pg_commit_ts, pg_logical added if wal strategy is set to
> sync.

Skipping some directories in a way similar to what a base backup does
would be nicer I think. We already have a list of those in
basebackup.c in the shape of excludeDirContents and excludeFiles. I
think that it would be a good idea to export those into a header that
pg_rewind could include, and refer to in order to exclude them when
fetching a set of files. At the end of the day, a rewind is a kind of
base backup in itself, and this patch would already serve well a lot
of people.

Having on top of that a way to exclude a wanted set of files and the
log directory (for example: should we look at log_directory and
exclude it from the fetched paths if it is not an absolute path?),
which is smart enough to take care of not removing paths critical for
a rewind like anything in base/, then you are good to go with a
full-blown tool that I think would serve the purposes you are looking
for.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] pg_rewind proposed scope and interface changes

2017-09-12 Thread Chris Travers
Hi all;


After working with pg_rewind a bit more it has become clear to me that some
aspects of the ideal scope of the program are a bit unclear.  This is in
part because the majority of cases where you use the utility, a small
amount of data loss is accepted due to the tradeoff of continuity.


Basic Responsibility:


pg_rewind’s core responsibility is to restore data directories so that they
are compatible with following another server.


Expanded Options:

As with pg_basebackup, wal files and related data may need to be specified
separately.


Pre-run checks:


PG_VERSION must be the same on target and source or else we stop.


We then check the timelines and if they have not diverged don’t do anything.


Processed directories:


pg_basebackup must *always* process global, base, and pg_tablespace
directories regardless of any WAL


Additionally the wal, xact, timestamp and logical directories must be
processed in some way.


  * if --wal=sync the directories are processed the way they are today

  * if --wal=clear then the contents of the directories are cleared and
replication is assumed to be used to bring the system up after.  Note this
will need to come with warning about the need for replication slots.


No other files will be clobbered.


This means that the basic processing file list will look like this:


Base, global, pg_tablespace


With

pg_wal, pg_xact, pg_commit_ts, pg_logical added if wal strategy is set to
sync.


Proposed patch roadmap:

   1. Improve WAL handling by allowing unlinking of all regular files in
   those directories during the rewind process.  Make —wal=sync the default
   for now because that is backwards compatible.  Longer-run we may consider
   switching the default.
   2. Restrict main operation to the directories listed above.

Problems that we will not try to solve:

 * Rewinding past table creation orphans table file.  This is a complex
topic on its own and probably needs a separate utility.

Thoughts?

-- 
Best Regards,
Chris Travers
Database Administrator

Tel: +49 162 9037 210 | Skype: einhverfr | www.adjust.com
Saarbrücker Straße 37a, 10405 Berlin