Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-07 Thread Michael Paquier
On Mon, Oct 07, 2019 at 03:31:45PM +0300, Alexey Kondratov wrote: > On 07.10.2019 4:06, Michael Paquier wrote: >> - --dry-run coverage is basically the same with the local and remote >> modes, so it seems like a waste of resource to run it for all the >> tests and all the modes. > > My point was

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-07 Thread Alexey Kondratov
On 07.10.2019 4:06, Michael Paquier wrote: On Fri, Oct 04, 2019 at 05:21:25PM +0300, Alexey Kondratov wrote: I've checked your patch, but it seems that it cannot be applied as is, since it e.g. adds a comment to 005_same_timeline.pl without actually changing the test. So I've slightly modified

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-06 Thread Michael Paquier
On Fri, Oct 04, 2019 at 05:21:25PM +0300, Alexey Kondratov wrote: > I've checked your patch, but it seems that it cannot be applied as is, since > it e.g. adds a comment to 005_same_timeline.pl without actually changing the > test. So I've slightly modified your patch and tried to fit both dry-run

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-04 Thread Alexey Kondratov
On 04.10.2019 11:37, Michael Paquier wrote: On Thu, Oct 03, 2019 at 12:43:37PM +0300, Alexey Kondratov wrote: On 03.10.2019 6:07, Michael Paquier wrote: I have reworked your first patch as per the attached. What do you think about it? The part with the control file needs to go down to v12,

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-04 Thread Michael Paquier
On Thu, Oct 03, 2019 at 12:43:37PM +0300, Alexey Kondratov wrote: > On 03.10.2019 6:07, Michael Paquier wrote: >> I have reworked your first patch as per the attached. What do you >> think about it? The part with the control file needs to go down to >> v12, and I would likely split that into two

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-03 Thread Alexey Kondratov
On 03.10.2019 6:07, Michael Paquier wrote: On Wed, Oct 02, 2019 at 08:28:09PM +0300, Alexey Kondratov wrote: I've directly followed your guess and tried to elaborate pg_rewind test cases and... It seems I've caught a few bugs: 1) --dry-run actually wasn't completely 'dry'. It did update target

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-02 Thread Michael Paquier
On Wed, Oct 02, 2019 at 08:28:09PM +0300, Alexey Kondratov wrote: > I've directly followed your guess and tried to elaborate pg_rewind test > cases and... It seems I've caught a few bugs: > > 1) --dry-run actually wasn't completely 'dry'. It did update target > controlfile, which could cause

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-10-02 Thread Alexey Kondratov
Hi Alvaro, On 30.09.2019 20:13, Alvaro Herrera wrote: OK, I pushed this patch as well as Alexey's test patch. It all works for me, and the coverage report shows that we're doing the new thing ... though only in the case that rewind *is* required. There is no test to verify the case where

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-30 Thread Paul Guo
> > BTW in the future if you have two separate patches, please post them in > separate threads and use separate commitfest items for each, even if > they have minor conflicts. > Sure. Thanks.

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-30 Thread Alvaro Herrera
On 2019-Mar-19, Paul Guo wrote: > Hello, Postgres hackers, > > Please see the attached patches. BTW in the future if you have two separate patches, please post them in separate threads and use separate commitfest items for each, even if they have minor conflicts. -- Álvaro Herrera

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-30 Thread Alvaro Herrera
OK, I pushed this patch as well as Alexey's test patch. It all works for me, and the coverage report shows that we're doing the new thing ... though only in the case that rewind *is* required. There is no test to verify the case where rewind is *not* required. I guess it'd also be good to test

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-30 Thread Alexey Kondratov
On 27.09.2019 17:28, Alvaro Herrera wrote: + # Now, when pg_rewind apparently succeeded with minimal permissions, + # add REPLICATION privilege. So we could test that new standby + # is able to connect to the new master with generated config. +

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-30 Thread Paul Guo
> > > I went through the remaining two patches and they seem to be very clear > and concise. However, there are two points I could complain about: > > 1) Maybe I've missed it somewhere in the thread above, but currently > pg_rewind allows to run itself with -R and --source-pgdata. In that case >

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-27 Thread Alvaro Herrera
On 2019-Sep-27, Paul Guo wrote: > I'm using --no-ensure-shutdown in the new version unless there are better > suggestions. That sounds sufficiently good. I pushed this patch, after fixing a few smallish problems, such as an assertion failure because of the terminating \n in the error message

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-27 Thread Alvaro Herrera
On 2019-Sep-27, Alexey Kondratov wrote: > 1) Maybe I've missed it somewhere in the thread above, but currently > pg_rewind allows to run itself with -R and --source-pgdata. In that case -R > option is just swallowed and neither standby.signal, nor > postgresql.auto.conf is written, which is

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-27 Thread Alexey Kondratov
On 27.09.2019 6:27, Paul Guo wrote: Secondarily, I see no reason to test connstr_source rather than just "conn" in the other patch; doing it the other way is more natural, since it's that thing that's tested as an argument. pg_rewind.c: Please put the new #include line

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-26 Thread Paul Guo
> > > > Note in the 2nd patch, the long option is changed as below. Both the > option > > and description > > now seems to be more concise since we want db state as either > DB_SHUTDOWNED > > or > > DB_SHUTDOWNED_IN_RECOVERY. > > > > "-s, --no-ensure-shutdowned do not auto-fix unclean

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-26 Thread Alvaro Herrera
> Thanks. I've updated the reset two patches and attached as v8. Great, thanks. > Note in the 2nd patch, the long option is changed as below. Both the option > and description > now seems to be more concise since we want db state as either DB_SHUTDOWNED > or > DB_SHUTDOWNED_IN_RECOVERY. > >

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-26 Thread Paul Guo
> > > Yes, -R is already used in pg_basebackup for the same functionality, so > it seems natural to use it here as well for consistency. > > I will review options naming in my own patch and update it accordingly. > Maybe -w/-W or -a/-A options will be good, since it's about WALs > retrieval from

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-26 Thread Paul Guo
On Thu, Sep 26, 2019 at 1:48 AM Alvaro Herrera wrote: > CC Alexey for reasons that become clear below. > > On 2019-Sep-25, Paul Guo wrote: > > > > Question about 0003. If we specify --skip-clean-shutdown and the > cluter > > > was not cleanly shut down, shouldn't we error out instead of trying

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-25 Thread a . kondratov
On 2019-09-25 20:48, Alvaro Herrera wrote: CC Alexey for reasons that become clear below. Another thing in 0002 is that you're adding a "-R" switch to pg_rewind, but we have another patch in the commitfest using the same switch for a different purpose. Maybe you guys need to get to an

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-25 Thread Laurenz Albe
On Wed, 2019-09-25 at 14:48 -0300, Alvaro Herrera wrote: > Another thing in 0002 is that you're adding a "-R" switch to > pg_rewind, but we have another patch in the commitfest using the same > switch for a different purpose. Maybe you guys need to get to an > agreement over who uses the letter

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-25 Thread Alvaro Herrera
CC Alexey for reasons that become clear below. On 2019-Sep-25, Paul Guo wrote: > > Question about 0003. If we specify --skip-clean-shutdown and the cluter > > was not cleanly shut down, shouldn't we error out instead of trying to > > press on? > > pg_rewind would error out in this case, see

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-24 Thread Paul Guo
> > On 2019-Sep-20, Paul Guo wrote: > > > The patch series failed on Windows CI. I modified the Windows build file > to > > fix that. See attached for the v7 version. > > Thanks. > > Question about 0003. If we specify --skip-clean-shutdown and the cluter > was not cleanly shut down, shouldn't we

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-24 Thread Alvaro Herrera
On 2019-Sep-20, Paul Guo wrote: > The patch series failed on Windows CI. I modified the Windows build file to > fix that. See attached for the v7 version. Thanks. Question about 0003. If we specify --skip-clean-shutdown and the cluter was not cleanly shut down, shouldn't we error out instead

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-20 Thread Paul Guo
The patch series failed on Windows CI. I modified the Windows build file to fix that. See attached for the v7 version. v7-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch Description: Binary data v7-0001-Extact-common-recovery-related-functions-from-pg_.patch Description: Binary

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-19 Thread Paul Guo
I've updated the patch series following the suggestions. Thanks. > > v6-0001-Extact-common-functions-from-pg_basebackup-into-s.patch Description: Binary data v6-0003-Ensure-target-clean-shutdown-at-the-beginning-of-.patch Description: Binary data

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-10 Thread Alvaro Herrera from 2ndQuadrant
On 2019-Sep-09, Paul Guo wrote: > > > > Thank for rebasing. > > > > I didn't like 0001 very much. > > > > * It seems now would be the time to stop pretending we're using a file > > called recovery.conf; I know we still support older server versions that > > use that file, but it sounds like we

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-09 Thread Paul Guo
> > Thank for rebasing. > > I didn't like 0001 very much. > > * It seems now would be the time to stop pretending we're using a file > called recovery.conf; I know we still support older server versions that > use that file, but it sounds like we should take the opportunity to > rename the

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-05 Thread Paul Guo
> > It seems there's minor breakage in the build, per CFbot. Can you > please rebase this? > > There is a code conflict. See attached for the new version. Thanks. v5-0001-Extact-common-functions-from-pg_basebackup-into-s.patch Description: Binary data

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-09-03 Thread Alvaro Herrera
On 2019-Jul-15, Paul Guo wrote: > Thanks. Both Mkvcbuild.pm and pg_rewind/Makefile are modified to make > Windows build pass in a > local environment (Hopefully this passes the CI testing), also now > pg_rewind/Makefile does not > create soft link for backup_common.h anymore. Instead -I is used

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-15 Thread Paul Guo
On Wed, Jul 10, 2019 at 3:28 PM Michael Paquier wrote: > On Tue, Jul 09, 2019 at 10:48:49PM +0800, Paul Guo wrote: > > Yes, the patches changed Makefile so that pg_rewind and pg_basebackup > could > > use some common code, but for Windows build, I'm not sure where are those > > window build

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-10 Thread Michael Paquier
On Tue, Jul 09, 2019 at 10:48:49PM +0800, Paul Guo wrote: > Yes, the patches changed Makefile so that pg_rewind and pg_basebackup could > use some common code, but for Windows build, I'm not sure where are those > window build files. Does anyone know about that? Thanks. The VS scripts are located

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-09 Thread Paul Guo
Yes, the patches changed Makefile so that pg_rewind and pg_basebackup could use some common code, but for Windows build, I'm not sure where are those window build files. Does anyone know about that? Thanks. On Tue, Jul 9, 2019 at 6:55 AM Thomas Munro wrote: > On Tue, Jul 2, 2019 at 5:46 PM Paul

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-08 Thread Thomas Munro
On Tue, Jul 2, 2019 at 5:46 PM Paul Guo wrote: > Rebased, aligned with recent changes in pg_rewind/pg_basebackup and then > retested. Thanks. Hi Paul, A minor build problem on Windows: src/bin/pg_rewind/pg_rewind.c(32): fatal error C1083: Cannot open include file: 'backup_common.h': No such

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-01 Thread Paul Guo
On Tue, Jul 2, 2019 at 12:35 AM Alvaro Herrera wrote: > On 2019-Apr-19, Paul Guo wrote: > > > The below patch runs single mode Postgres if needed to make sure the > target > > is cleanly shutdown. A new option is added (off by default). > >

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-01 Thread Paul Guo
Rebased, aligned with recent changes in pg_rewind/pg_basebackup and then retested. Thanks. On Mon, Jul 1, 2019 at 7:35 PM Thomas Munro wrote: > On Fri, Apr 19, 2019 at 3:41 PM Paul Guo wrote: > > I updated the patches as attached following previous discussions. > > Hi Paul, > > Could we please

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-01 Thread Alvaro Herrera
On 2019-Apr-19, Paul Guo wrote: > The below patch runs single mode Postgres if needed to make sure the target > is cleanly shutdown. A new option is added (off by default). > v2-0001-Ensure-target-clean-shutdown-at-beginning-of-pg_r.patch Why do we need an option for this? Is there a reason not

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-07-01 Thread Thomas Munro
On Fri, Apr 19, 2019 at 3:41 PM Paul Guo wrote: > I updated the patches as attached following previous discussions. Hi Paul, Could we please have a fresh rebase now that the CF is here? Thanks, -- Thomas Munro https://enterprisedb.com

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-04-18 Thread Paul Guo
Hi Michael, I updated the patches as attached following previous discussions. The two patches: v2-0001-Extact-common-functions-from-pg_basebackup-into-s.patch v2-0002-Add-option-to-write-recovery-configuration-inform.patch 1. 0001 does move those common functions & variables to two new files

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Wed, Mar 20, 2019 at 1:20 PM Michael Paquier wrote: > On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: > > This is a good suggestion also. Will do it. > > Please note also that we don't care about recovery.conf since v12 as > recovery parameters are now GUCs. I would suggest

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Michael Paquier
On Wed, Mar 20, 2019 at 12:48:52PM +0800, Paul Guo wrote: > This is a good suggestion also. Will do it. Please note also that we don't care about recovery.conf since v12 as recovery parameters are now GUCs. I would suggest appending those extra parameters to postgresql.auto.conf, which is what

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
On Tue, Mar 19, 2019 at 2:18 PM Michael Paquier wrote: > On Tue, Mar 19, 2019 at 02:09:03PM +0800, Paul Guo wrote: > > The first patch adds an option to automatically generate recovery conf > > contents in related files, following pg_basebackup. In the patch, > > GenerateRecoveryConf(),

Re: Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Michael Paquier
On Tue, Mar 19, 2019 at 02:09:03PM +0800, Paul Guo wrote: > The first patch adds an option to automatically generate recovery conf > contents in related files, following pg_basebackup. In the patch, > GenerateRecoveryConf(), WriteRecoveryConf() and escape_quotes() are almost > same as them on

Two pg_rewind patches (auto generate recovery conf and ensure clean shutdown)

2019-03-19 Thread Paul Guo
Hello, Postgres hackers, Please see the attached patches. The first patch adds an option to automatically generate recovery conf contents in related files, following pg_basebackup. In the patch, GenerateRecoveryConf(), WriteRecoveryConf() and escape_quotes() are almost same as them on