Re: [HACKERS] Proposal: pg_rewind to skip config files

On Thu, Sep 7, 2017 at 2:47 PM, Alvaro Herrera wrote: > After reading this discussion, I agree that pg_rewind needs to become > smarter in order to be fully useful in production environments; right > now there are many warts and corner cases that did not seem to have been > considered during the

Re: [HACKERS] Proposal: pg_rewind to skip config files

After reading this discussion, I agree that pg_rewind needs to become smarter in order to be fully useful in production environments; right now there are many warts and corner cases that did not seem to have been considered during the initial development (which I think is all right, taking into acc

Re: [HACKERS] Proposal: pg_rewind to skip config files

One more side to this which is relevant to other discussions. If I am rewinding back to before when a table was created, the current algorithm as well as any proposed algorithms will delete the reference to the relfilenode in the catalogs but not the file itself. I don't see how an undo subsystem

Re: [HACKERS] Proposal: pg_rewind to skip config files

> 5 сент. 2017 г., в 15:48, Michael Paquier > написал(а): > > On Tue, Sep 5, 2017 at 9:40 PM, Vladimir Borodin wrote: >> We do compress WALs and send them over network. Doing it via archive_command >> in single thread is sometimes slower than new WALs are written under heavy >> load. > > Ah,

Re: [HACKERS] Proposal: pg_rewind to skip config files

> 5 сент. 2017 г., в 15:42, Chris Travers написал(а): > > On Tue, Sep 5, 2017 at 2:40 PM, Vladimir Borodin > wrote: > >> 5 сент. 2017 г., в 14:04, Michael Paquier > > написал(а): >> >>> For example, in archive_command we put WALs for

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Tue, Sep 5, 2017 at 9:40 PM, Vladimir Borodin wrote: > We do compress WALs and send them over network. Doing it via archive_command > in single thread is sometimes slower than new WALs are written under heavy > load. Ah, yeah, true. I do use pg_receivexlog --compress for that locally and do a

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Tue, Sep 5, 2017 at 2:40 PM, Vladimir Borodin wrote: > > 5 сент. 2017 г., в 14:04, Michael Paquier > написал(а): > > For example, in archive_command we put WALs for archiving from > pg_xlog/pg_wal into another directory inside PGDATA and than another cron > task makes real archiving. This dir

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Tue, Sep 5, 2017 at 1:04 PM, Michael Paquier wrote: > On Tue, Sep 5, 2017 at 7:54 PM, Vladimir Borodin wrote: > > 5 сент. 2017 г., в 12:31, Chris Travers > > написал(а): > > > > I think the simplest solution for now is to skip any files ending in > .conf, > > .log, and serverlog. > > This is

Re: [HACKERS] Proposal: pg_rewind to skip config files

> 5 сент. 2017 г., в 14:04, Michael Paquier > написал(а): > >> For example, in archive_command we put WALs for archiving from >> pg_xlog/pg_wal into another directory inside PGDATA and than another cron >> task makes real archiving. This directory ideally should be skipped by >> pg_rewind, but

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Tue, Sep 5, 2017 at 12:54 PM, Vladimir Borodin wrote: > > 5 сент. 2017 г., в 12:31, Chris Travers > написал(а): > > I think the simplest solution for now is to skip any files ending in > .conf, .log, and serverlog. > > > Why don’t you want to solve the problem once? It is a bit harder to get

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Mon, Sep 4, 2017 at 10:38 PM, Alvaro Herrera wrote: > I wonder how portable fnmatch() is in practice (which we don't currently > use anywhere). A shell glob seems a more natural interface to me for > this than a regular expression. On Windows you could use roughly PathMatchSpecEx, but it does

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Tue, Sep 5, 2017 at 7:54 PM, Vladimir Borodin wrote: > 5 сент. 2017 г., в 12:31, Chris Travers > написал(а): > > I think the simplest solution for now is to skip any files ending in .conf, > .log, and serverlog. This is not a portable solution. Users can include configuration files with the n

Re: [HACKERS] Proposal: pg_rewind to skip config files

> 5 сент. 2017 г., в 12:31, Chris Travers написал(а): > > I think the simplest solution for now is to skip any files ending in .conf, > .log, and serverlog. Why don’t you want to solve the problem once? It is a bit harder to get consensus on a way how to do it, but it seems that there are no

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Mon, Sep 4, 2017 at 3:38 PM, Alvaro Herrera wrote: > Chris Travers wrote: > > On Mon, Sep 4, 2017 at 12:23 PM, Michael Paquier < > michael.paqu...@gmail.com> > > wrote: > > > > > On Mon, Sep 4, 2017 at 7:21 PM, Michael Paquier > > > wrote: > > > > A simple idea would be to pass as a parameter

Re: [HACKERS] Proposal: pg_rewind to skip config files

Chris Travers wrote: > On Mon, Sep 4, 2017 at 12:23 PM, Michael Paquier > wrote: > > > On Mon, Sep 4, 2017 at 7:21 PM, Michael Paquier > > wrote: > > > A simple idea would be to pass as a parameter a regex on which we > > > check files to skip when scanning the directory of the target remotely >

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Mon, Sep 4, 2017 at 12:23 PM, Michael Paquier wrote: > On Mon, Sep 4, 2017 at 7:21 PM, Michael Paquier > wrote: > > A simple idea would be to pass as a parameter a regex on which we > > check files to skip when scanning the directory of the target remotely > > or locally. This needs to be use

Re: [HACKERS] Proposal: pg_rewind to skip config files

Ok so I have a proof of concept patch here. This is proof of concept only. It odes not change documentation or the like. The purpose of the patch is discussion on the "do we want this" side. The patch is fairly trivial but I have not added test cases or changed docs yet. Intention of the patch

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Mon, Sep 4, 2017 at 7:21 PM, Michael Paquier wrote: > A simple idea would be to pass as a parameter a regex on which we > check files to skip when scanning the directory of the target remotely > or locally. This needs to be used with care though, it would be easy > to corrupt an instance. I ac

Re: [HACKERS] Proposal: pg_rewind to skip config files

On Mon, Sep 4, 2017 at 5:53 PM, Chris Travers wrote: > In some experiments with pg_rewind and rep mgr I noticed that local testing > is complicated by the fact that pg_rewind appears to copy configuration > files from the source to target directory. > > I would propose to make a modest patch to ex

Re: [HACKERS] Proposal: pg_rewind to skip config files

On 2017-09-04 11:53, Chris Travers wrote: In some experiments with pg_rewind and rep mgr I noticed that local testing is complicated by the fact that pg_rewind appears to copy configuration files from the source to target directory. I would propose to make a modest patch to exclude postgresql.co