Re: speed up a logical replica setup

2024-05-24 Thread Shlok Kyal
On Wed, 22 May 2024 at 16:50, Amit Kapila wrote: > > On Mon, May 20, 2024 at 4:30 PM Shlok Kyal wrote: > > > > > > I was trying to test this utility when 'sync_replication_slots' is on > > > and it gets in an ERROR loop [1] and never finishes. Please find the > > > postgresql.auto used on the

Re: speed up a logical replica setup

2024-05-23 Thread Amit Kapila
On Thu, May 23, 2024 at 8:43 PM Euler Taveira wrote: > > On Thu, May 23, 2024, at 5:54 AM, Amit Kapila wrote: > > > Why in the first place do we need to ensure that primary_slot_name is > active on the primary? You mentioned something related to WAL > retention but I don't know how that is

Re: speed up a logical replica setup

2024-05-23 Thread Euler Taveira
On Thu, May 23, 2024, at 5:54 AM, Amit Kapila wrote: > On Wed, May 22, 2024 at 8:46 PM Euler Taveira wrote: > > > > Following the same line that simplifies the code, we can: (a) add a loop in > > check_subscriber() that waits until walreceiver is available on subscriber > > or > > (b) use a

Re: speed up a logical replica setup

2024-05-23 Thread Amit Kapila
On Wed, May 22, 2024 at 8:46 PM Euler Taveira wrote: > > On Wed, May 22, 2024, at 8:19 AM, Amit Kapila wrote: > > > v2-0002: not changed > > > > We have added more tries to see if the primary_slot_name becomes > active but I think it is still fragile because it is possible on slow > machines that

Re: speed up a logical replica setup

2024-05-22 Thread Euler Taveira
On Wed, May 22, 2024, at 8:19 AM, Amit Kapila wrote: > > > > v2-0001: not changed > > > > Shouldn't we modify it as per the suggestion given in the email [1]? I > am wondering if we can entirely get rid of checking the primary > business and simply rely on recovery_timeout and keep checking >

Re: speed up a logical replica setup

2024-05-22 Thread Amit Kapila
On Mon, May 20, 2024 at 4:30 PM Shlok Kyal wrote: > > > > I was trying to test this utility when 'sync_replication_slots' is on > > and it gets in an ERROR loop [1] and never finishes. Please find the > > postgresql.auto used on the standby attached. I think if the standby > > has enabled

Re: speed up a logical replica setup

2024-05-20 Thread Shlok Kyal
Hi, > > I was trying to test this utility when 'sync_replication_slots' is on > and it gets in an ERROR loop [1] and never finishes. Please find the > postgresql.auto used on the standby attached. I think if the standby > has enabled sync_slots, you need to pass dbname in >

Re: speed up a logical replica setup

2024-05-20 Thread Amit Kapila
On Sun, May 19, 2024 at 4:25 AM Thomas Munro wrote: > > 040_pg_createsubscriber.pl seems to be failing occasionally on > culicidae near "--dry-run on node S". I couldn't immediately see why. > That animal is using EXEC_BACKEND and I also saw a one-off failure a > bit like that on my own local

Re: speed up a logical replica setup

2024-05-18 Thread Thomas Munro
040_pg_createsubscriber.pl seems to be failing occasionally on culicidae near "--dry-run on node S". I couldn't immediately see why. That animal is using EXEC_BACKEND and I also saw a one-off failure a bit like that on my own local Linux + EXEC_BACKEND test run (sorry I didn't keep the details

Re: speed up a logical replica setup

2024-04-30 Thread Amit Kapila
On Tue, Apr 30, 2024 at 12:04 PM Amit Kapila wrote: > > On Mon, Apr 29, 2024 at 5:28 PM Amit Kapila wrote: > > > > On Mon, Apr 29, 2024 at 5:23 PM Euler Taveira wrote: > > > > I was trying to test this utility when 'sync_replication_slots' is on > and it gets in an ERROR loop [1] and never

Re: speed up a logical replica setup

2024-04-30 Thread Amit Kapila
On Mon, Apr 29, 2024 at 5:28 PM Amit Kapila wrote: > > On Mon, Apr 29, 2024 at 5:23 PM Euler Taveira wrote: > I was trying to test this utility when 'sync_replication_slots' is on and it gets in an ERROR loop [1] and never finishes. Please find the postgresql.auto used on the standby attached.

Re: speed up a logical replica setup

2024-04-29 Thread Amit Kapila
On Mon, Apr 29, 2024 at 5:23 PM Euler Taveira wrote: > > On Mon, Apr 29, 2024, at 6:56 AM, Amit Kapila wrote: > > On Wed, Mar 27, 2024 at 1:47 AM Euler Taveira wrote: > > > > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > > > > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS

Re: speed up a logical replica setup

2024-04-29 Thread Euler Taveira
On Mon, Apr 29, 2024, at 6:56 AM, Amit Kapila wrote: > On Wed, Mar 27, 2024 at 1:47 AM Euler Taveira wrote: > > > > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > > > > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? > > Why isn't recovery_timeout enough to

Re: speed up a logical replica setup

2024-04-29 Thread Amit Kapila
On Tue, Mar 26, 2024 at 8:24 AM Euler Taveira wrote: > > On Mon, Mar 25, 2024, at 1:06 PM, Hayato Kuroda (Fujitsu) wrote: > > The first patch implements a combination of (1) and (2). > > ## Analysis for failure 2 > > According to [2], the physical replication slot which is specified as >

Re: speed up a logical replica setup

2024-04-29 Thread Amit Kapila
On Wed, Mar 27, 2024 at 1:47 AM Euler Taveira wrote: > > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? > Why isn't recovery_timeout enough to decide if wait_for_end_recovery() > waited long enough? > > > It was

Re: speed up a logical replica setup

2024-03-26 Thread Tomas Vondra
On 3/26/24 21:17, Euler Taveira wrote: > On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: >> Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? >> Why isn't recovery_timeout enough to decide if wait_for_end_recovery() >> waited long enough? > > It was an attempt to

Re: speed up a logical replica setup

2024-03-26 Thread Euler Taveira
On Tue, Mar 26, 2024, at 4:12 PM, Tomas Vondra wrote: > Perhaps I'm missing something, but why is NUM_CONN_ATTEMPTS even needed? > Why isn't recovery_timeout enough to decide if wait_for_end_recovery() > waited long enough? It was an attempt to decoupled a connection failure (that keeps streaming

Re: speed up a logical replica setup

2024-03-26 Thread Tomas Vondra
On 3/26/24 03:53, Euler Taveira wrote: > On Mon, Mar 25, 2024, at 1:06 PM, Hayato Kuroda (Fujitsu) wrote: >> ## Analysis for failure 1 >> >> The failure caused by a time lag between walreceiver finishes and >> pg_is_in_recovery() >> returns true. >> >> According to the output [1], it seems that

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
Dear Amit, Euler, > > This only drops the publications created by this tool, not the > pre-existing ones that we discussed in the link provided. Another concern around here is the case which primary subscribes changes from others. After the conversion, new subscriber also tries to connect to

Re: speed up a logical replica setup

2024-03-25 Thread Amit Kapila
On Tue, Mar 26, 2024 at 8:27 AM Euler Taveira wrote: > > On Mon, Mar 25, 2024, at 11:33 PM, Amit Kapila wrote: > > On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > > > I have committed your version v33. I did another pass over the > > identifier and literal quoting. I added quoting

Re: speed up a logical replica setup

2024-03-25 Thread Euler Taveira
On Mon, Mar 25, 2024, at 11:33 PM, Amit Kapila wrote: > On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > > > I have committed your version v33. I did another pass over the > > identifier and literal quoting. I added quoting for replication slot > > names, for example, even though

Re: speed up a logical replica setup

2024-03-25 Thread Euler Taveira
On Mon, Mar 25, 2024, at 1:06 PM, Hayato Kuroda (Fujitsu) wrote: > ## Analysis for failure 1 > > The failure caused by a time lag between walreceiver finishes and > pg_is_in_recovery() > returns true. > > According to the output [1], it seems that the tool failed at > wait_for_end_recovery() >

Re: speed up a logical replica setup

2024-03-25 Thread Amit Kapila
On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > I have committed your version v33. I did another pass over the > identifier and literal quoting. I added quoting for replication slot > names, for example, even though they can only contain a restricted set > of characters, but it felt

Re: speed up a logical replica setup

2024-03-25 Thread vignesh C
On Mon, 25 Mar 2024 at 21:36, Hayato Kuroda (Fujitsu) wrote: > > Dear Bharath, Peter, > > > Looks like BF animals aren't happy, please check - > > > https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. > > > > Looks like sanitizer failures. There were a few messages about that > >

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
Dear Bharath, Peter, > Looks like BF animals aren't happy, please check - > > https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. > > Looks like sanitizer failures. There were a few messages about that > recently, but those were all just about freeing memory after use, which > we don't

Re: speed up a logical replica setup

2024-03-25 Thread Euler Taveira
On Mon, Mar 25, 2024, at 8:55 AM, Peter Eisentraut wrote: > On 22.03.24 04:31, Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: > >> There is a compilation error while building postgres with the patch > >> due to a recent commit. I have attached a top-up patch v32-0003

Re: speed up a logical replica setup

2024-03-25 Thread Peter Eisentraut
On 25.03.24 13:36, Bharath Rupireddy wrote: On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: I have committed your version v33. Looks like BF animals aren't happy, please check - https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. Looks like sanitizer failures. There were a

Re: speed up a logical replica setup

2024-03-25 Thread Bharath Rupireddy
On Mon, Mar 25, 2024 at 5:25 PM Peter Eisentraut wrote: > > I have committed your version v33. Looks like BF animals aren't happy, please check - https://buildfarm.postgresql.org/cgi-bin/show_failures.pl. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web

Re: speed up a logical replica setup

2024-03-25 Thread Peter Eisentraut
On 22.03.24 04:31, Euler Taveira wrote: On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: There is a compilation error while building postgres with the patch due to a recent commit. I have attached a top-up patch v32-0003 to resolve this compilation error. I have not updated the version of

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
> IIUC, added options were inspired by Tomas. And he said the limitation > (pub/sub/slot > name cannot be specified) was acceptable for the first version. I agree with > him. > (To be honest, I feel that options should be fewer for the first release) Just to confirm - I don't think it is not

RE: speed up a logical replica setup

2024-03-25 Thread Hayato Kuroda (Fujitsu)
Dear Amit, Euler, I also want to share my opinion, just in case. > On Thu, Mar 21, 2024 at 8:00 PM Euler Taveira wrote: > > > > On Thu, Mar 21, 2024, at 10:33 AM, vignesh C wrote: > > > > If we commit this we might not be able to change the way the option > > behaves once the customers starts

Re: speed up a logical replica setup

2024-03-22 Thread Shubham Khanna
On Fri, Mar 22, 2024 at 9:02 AM Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: > > There is a compilation error while building postgres with the patch > due to a recent commit. I have attached a top-up patch v32-0003 to > resolve this compilation error. > I have not

Re: speed up a logical replica setup

2024-03-22 Thread vignesh C
On Fri, 22 Mar 2024 at 09:01, Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: > > There is a compilation error while building postgres with the patch > due to a recent commit. I have attached a top-up patch v32-0003 to > resolve this compilation error. > I have not

Re: speed up a logical replica setup

2024-03-21 Thread Amit Kapila
On Fri, Mar 22, 2024 at 9:44 AM Fabrízio de Royes Mello wrote: > > On Fri, Mar 22, 2024 at 12:54 AM Amit Kapila wrote: > > > > > > The user choosing to convert a physical standby to a subscriber would > > in some cases be interested in converting it for all the databases > > (say for the case of

Re: speed up a logical replica setup

2024-03-21 Thread Fabrízio de Royes Mello
On Fri, Mar 22, 2024 at 12:54 AM Amit Kapila wrote: > > > The user choosing to convert a physical standby to a subscriber would > in some cases be interested in converting it for all the databases > (say for the case of upgrade [1]) and giving options for each database > would be cumbersome for

Re: speed up a logical replica setup

2024-03-21 Thread Amit Kapila
On Thu, Mar 21, 2024 at 8:00 PM Euler Taveira wrote: > > On Thu, Mar 21, 2024, at 10:33 AM, vignesh C wrote: > > If we commit this we might not be able to change the way the option > behaves once the customers starts using it. How about removing these > options in the first version and adding it

Re: speed up a logical replica setup

2024-03-21 Thread Euler Taveira
On Thu, Mar 21, 2024, at 6:49 AM, Shlok Kyal wrote: > There is a compilation error while building postgres with the patch > due to a recent commit. I have attached a top-up patch v32-0003 to > resolve this compilation error. > I have not updated the version of the patch as I have not made any >

Re: speed up a logical replica setup

2024-03-21 Thread Euler Taveira
On Thu, Mar 21, 2024, at 10:33 AM, vignesh C wrote: > If we commit this we might not be able to change the way the option > behaves once the customers starts using it. How about removing these > options in the first version and adding it in the next version after > more discussion. We don't need

Re: speed up a logical replica setup

2024-03-21 Thread vignesh C
On Thu, 21 Mar 2024 at 18:02, Peter Eisentraut wrote: > > On 21.03.24 12:35, vignesh C wrote: > > Here are a few suggestions: > > 1) I felt displaying the server log to the console is not a good idea, > > I prefer this to be logged. There were similar comments from > > Kuroda-san at [1], Peter at

Re: speed up a logical replica setup

2024-03-21 Thread Peter Eisentraut
On 21.03.24 12:35, vignesh C wrote: Here are a few suggestions: 1) I felt displaying the server log to the console is not a good idea, I prefer this to be logged. There were similar comments from Kuroda-san at [1], Peter at [2]. The number of lines will increase based on the log level set. If

Re: speed up a logical replica setup

2024-03-21 Thread vignesh C
On Thu, 21 Mar 2024 at 09:50, Euler Taveira wrote: > > On Mon, Mar 18, 2024, at 10:52 AM, Peter Eisentraut wrote: > > On 16.03.24 16:42, Euler Taveira wrote: > > I'm attaching a new version (v30) that adds: > > I have some review comments and attached a patch with some smaller > fixups (mainly

Re: speed up a logical replica setup

2024-03-20 Thread Euler Taveira
On Wed, Mar 20, 2024, at 7:16 AM, Shubham Khanna wrote: > On Tue, Mar 19, 2024 at 8:54 PM vignesh C wrote: > > > > If you are not planning to have the checks for name length, this could > > alternatively be fixed by including database id also while querying > > pg_subscription like below in

Re: speed up a logical replica setup

2024-03-20 Thread Euler Taveira
On Tue, Mar 19, 2024, at 8:57 AM, Peter Eisentraut wrote: > On 19.03.24 12:26, Shlok Kyal wrote: > > > > I have added a top-up patch v30-0003. The issue in [1] still exists in > > the v30 patch. I was not able to come up with an approach to handle it > > in the code, so I have added it to the

Re: speed up a logical replica setup

2024-03-20 Thread Euler Taveira
On Mon, Mar 18, 2024, at 2:43 AM, Hayato Kuroda (Fujitsu) wrote: > Thanks for updating the patch. Here are my comments. > I used Grammarly to proofread sentences. > (The tool strongly recommends to use active voice, but I can ignore for now) Thanks for another review. I posted a new patch (v32)

Re: speed up a logical replica setup

2024-03-20 Thread Euler Taveira
On Mon, Mar 18, 2024, at 10:52 AM, Peter Eisentraut wrote: > On 16.03.24 16:42, Euler Taveira wrote: > > I'm attaching a new version (v30) that adds: > > I have some review comments and attached a patch with some smaller > fixups (mainly message wording and avoid fixed-size string buffers).

Re: speed up a logical replica setup

2024-03-20 Thread Shlok Kyal
> Hi, > > > I'm attaching a new version (v30) that adds: > > > > * 3 new options (--publication, --subscription, --replication-slot) to > > assign > > names to the objects. The --database option used to ignore duplicate > > names, > > however, since these new options rely on the number of

Re: speed up a logical replica setup

2024-03-19 Thread Peter Eisentraut
On 19.03.24 16:24, vignesh C wrote: The problem with this failure is that standby has been promoted already and we will have to re-create the physica replica again. If you are not planning to have the checks for name length, this could alternatively be fixed by including database id also while

Re: speed up a logical replica setup

2024-03-19 Thread vignesh C
On Mon, 18 Mar 2024 at 16:36, Peter Eisentraut wrote: > > On 18.03.24 08:18, vignesh C wrote: > > 1) Maximum size of the object name is 64, we can have a check so that > > we don't specify more than the maximum allowed length: > > + case 3: > > + if (!simple_string_list_member(_names, optarg)) >

Re: speed up a logical replica setup

2024-03-19 Thread Peter Eisentraut
On 19.03.24 12:26, Shlok Kyal wrote: I'm attaching a new version (v30) that adds: * 3 new options (--publication, --subscription, --replication-slot) to assign names to the objects. The --database option used to ignore duplicate names, however, since these new options rely on the number

Re: speed up a logical replica setup

2024-03-19 Thread Peter Eisentraut
On 19.03.24 08:05, Amit Kapila wrote: On Mon, Mar 18, 2024 at 7:22 PM Peter Eisentraut wrote: In check_subscriber(): All these permissions checks seem problematic to me. We shouldn't reimplement our own copy of the server's permission checks. The server can check the permissions. And if

Re: speed up a logical replica setup

2024-03-19 Thread Shlok Kyal
Hi, > I'm attaching a new version (v30) that adds: > > * 3 new options (--publication, --subscription, --replication-slot) to assign > names to the objects. The --database option used to ignore duplicate names, > however, since these new options rely on the number of database options to >

Re: speed up a logical replica setup

2024-03-19 Thread Amit Kapila
On Mon, Mar 18, 2024 at 7:22 PM Peter Eisentraut wrote: > > In check_subscriber(): All these permissions checks seem problematic > to me. We shouldn't reimplement our own copy of the server's > permission checks. The server can check the permissions. And if the > permission checking in the

RE: speed up a logical replica setup

2024-03-18 Thread Hayato Kuroda (Fujitsu)
Dear Peter, Thanks for giving comments. I want to reply some of them. > > 17. > > > > "specifies promote" > > > > We can do double-quote for the word promote. > > The v30 patch has promote, which I think is adequate. Opps. Actually I did look v29 patch while firstly reviewing. Sorry for noise.

Re: speed up a logical replica setup

2024-03-18 Thread Alvaro Herrera
On 2024-Mar-18, Peter Eisentraut wrote: > * src/bin/pg_basebackup/pg_createsubscriber.c > > I think the connection string handling is not robust against funny > characters, like spaces, in database names etc. Maybe it would be easier to deal with this by passing around a struct with

Re: speed up a logical replica setup

2024-03-18 Thread Peter Eisentraut
On 16.03.24 16:42, Euler Taveira wrote: I'm attaching a new version (v30) that adds: I have some review comments and attached a patch with some smaller fixups (mainly message wording and avoid fixed-size string buffers). * doc/src/sgml/ref/pg_createsubscriber.sgml I would remove the "How

Re: speed up a logical replica setup

2024-03-18 Thread Peter Eisentraut
On 18.03.24 06:43, Hayato Kuroda (Fujitsu) wrote: 02. "The main difference between the logical replication setup and pg_createsubscriber is the initial data copy." Grammarly suggests: "The initial data copy is the main difference between the logical replication setup and pg_createsubscriber."

Re: speed up a logical replica setup

2024-03-18 Thread Peter Eisentraut
On 18.03.24 08:18, vignesh C wrote: 1) Maximum size of the object name is 64, we can have a check so that we don't specify more than the maximum allowed length: + case 3: + if (!simple_string_list_member(_names, optarg)) + { + simple_string_list_append(_names, optarg); + num_replslots++; + } +

Re: speed up a logical replica setup

2024-03-18 Thread vignesh C
On Sat, 16 Mar 2024 at 21:13, Euler Taveira wrote: > > On Fri, Mar 15, 2024, at 3:34 AM, Amit Kapila wrote: > > Did you consider adding options for publication/subscription/slot > names as mentioned in my previous email? As discussed in a few emails > above, it would be quite confusing for users

RE: speed up a logical replica setup

2024-03-17 Thread Hayato Kuroda (Fujitsu)
Dear Euler, Thanks for updating the patch. Here are my comments. I used Grammarly to proofread sentences. (The tool strongly recommends to use active voice, but I can ignore for now) 01. "After a successful run, the state of the target server is analagous to a fresh logical replication setup."

Re: speed up a logical replica setup

2024-03-17 Thread Amit Kapila
On Mon, Mar 18, 2024 at 9:37 AM Amit Kapila wrote: > > On Sat, Mar 16, 2024 at 9:13 PM Euler Taveira wrote: > > > > On Fri, Mar 15, 2024, at 3:34 AM, Amit Kapila wrote: > > > > Did you consider adding options for publication/subscription/slot > > names as mentioned in my previous email? As

Re: speed up a logical replica setup

2024-03-17 Thread Amit Kapila
On Sat, Mar 16, 2024 at 9:13 PM Euler Taveira wrote: > > On Fri, Mar 15, 2024, at 3:34 AM, Amit Kapila wrote: > > Did you consider adding options for publication/subscription/slot > names as mentioned in my previous email? As discussed in a few emails > above, it would be quite confusing for

Re: speed up a logical replica setup

2024-03-16 Thread vignesh C
On Sat, 16 Mar 2024 at 21:16, Euler Taveira wrote: > > On Sat, Mar 16, 2024, at 10:31 AM, vignesh C wrote: > > I was able to reproduce this random failure and found the following reason: > The Minimum recovery ending location 0/500 was more than the > recovery_target_lsn specified is

Re: speed up a logical replica setup

2024-03-16 Thread Euler Taveira
On Sat, Mar 16, 2024, at 10:31 AM, vignesh C wrote: > I was able to reproduce this random failure and found the following reason: > The Minimum recovery ending location 0/500 was more than the > recovery_target_lsn specified is "0/4001198". In few random cases the > standby applies a few more

Re: speed up a logical replica setup

2024-03-16 Thread Euler Taveira
On Fri, Mar 15, 2024, at 3:34 AM, Amit Kapila wrote: > Did you consider adding options for publication/subscription/slot > names as mentioned in my previous email? As discussed in a few emails > above, it would be quite confusing for users to identify the logical > replication objects once the

Re: speed up a logical replica setup

2024-03-16 Thread vignesh C
On Mon, 11 Mar 2024 at 10:33, Hayato Kuroda (Fujitsu) wrote: > > Dear Vignesh, > > Thanks for updating the patch, but cfbot still got angry [1]. > Note that two containers (autoconf and meson) failed at different place, > so I think it is intentional one. It seems that there may be a bug related

Re: speed up a logical replica setup

2024-03-15 Thread Amit Kapila
On Fri, Mar 15, 2024 at 9:23 AM Euler Taveira wrote: > Did you consider adding options for publication/subscription/slot names as mentioned in my previous email? As discussed in a few emails above, it would be quite confusing for users to identify the logical replication objects once the standby

Re: speed up a logical replica setup

2024-03-14 Thread Euler Taveira
On Wed, Mar 13, 2024, at 10:09 AM, Shlok Kyal wrote: > Added a top-up patch v28-0005 to fix this issue. > I am not changing the version as v28-0001 to v28-0004 is the same as above. Thanks for your review! I'm posting a new patch (v29) that merges the previous patches (v28-0002 and v28-0003). I

Re: speed up a logical replica setup

2024-03-13 Thread Euler Taveira
On Fri, Mar 8, 2024, at 6:44 AM, Hayato Kuroda (Fujitsu) wrote: > Hmm, right. I considered below improvements. Tomas and Euler, how do you > think? I'm posting a new patchset v28. I changed the way that the check function works. From the usability perspective, it is better to test all

Re: speed up a logical replica setup

2024-03-12 Thread Amit Kapila
On Mon, Mar 11, 2024 at 9:42 AM vignesh C wrote: > > On Sat, 9 Mar 2024 at 00:56, Tomas Vondra > wrote: > > > > >> 5) slot / publication / subscription name > > >> > > >> I find it somewhat annoying it's not possible to specify names for > > >> objects created by the tool - replication slots,

Re: speed up a logical replica setup

2024-03-12 Thread Amit Kapila
On Thu, Mar 7, 2024 at 10:44 PM Tomas Vondra wrote: > > 4) Is FOR ALL TABLES a good idea? > > I'm not sure FOR ALL TABLES is a good idea. Or said differently, I'm > sure it won't work for a number of use cases. I know large databases > it's common to create "work tables" (not necessarily

RE: speed up a logical replica setup

2024-03-10 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, Thanks for updating the patch, but cfbot still got angry [1]. Note that two containers (autoconf and meson) failed at different place, so I think it is intentional one. It seems that there may be a bug related with 32-bit build. We should see and fix as soon as possible. [1]:

Re: speed up a logical replica setup

2024-03-10 Thread vignesh C
On Sat, 9 Mar 2024 at 00:56, Tomas Vondra wrote: > > > > On 3/8/24 10:44, Hayato Kuroda (Fujitsu) wrote: > > Dear Tomas, Euler, > > > > Thanks for starting to read the thread! Since I'm not an original author, > > I want to reply partially. > > > >> I decided to take a quick look on this patch

Re: speed up a logical replica setup

2024-03-08 Thread Tomas Vondra
On 3/8/24 10:44, Hayato Kuroda (Fujitsu) wrote: > Dear Tomas, Euler, > > Thanks for starting to read the thread! Since I'm not an original author, > I want to reply partially. > >> I decided to take a quick look on this patch today, to see how it works >> and do some simple tests. I've only

Re: speed up a logical replica setup

2024-03-08 Thread Andrey M. Borodin
> On 8 Mar 2024, at 12:03, Shlok Kyal wrote: > > I haven't digged into the thread, but recent version fails some CFbot's tests. http://commitfest.cputube.org/euler-taveira.html https://cirrus-ci.com/task/4833499115421696 ==29928==ERROR: AddressSanitizer: heap-use-after-free on address

RE: speed up a logical replica setup

2024-03-08 Thread Hayato Kuroda (Fujitsu)
Dear Tomas, Euler, Thanks for starting to read the thread! Since I'm not an original author, I want to reply partially. > I decided to take a quick look on this patch today, to see how it works > and do some simple tests. I've only started to get familiar with it, so > I have only some comments

Re: speed up a logical replica setup

2024-03-08 Thread vignesh C
On Thu, 7 Mar 2024 at 18:31, Shlok Kyal wrote: > > Hi, > > > Thanks for the feedback. I'm attaching v26 that addresses most of your > > comments > > and some issues pointed by Vignesh [1]. > > I have created a top-up patch v27-0004. It contains additional test > cases for pg_createsubscriber. >

Re: speed up a logical replica setup

2024-03-07 Thread Tomas Vondra
Hi, I decided to take a quick look on this patch today, to see how it works and do some simple tests. I've only started to get familiar with it, so I have only some comments / questions regarding usage, not on the code. It's quite possible I didn't understand some finer points, or maybe it was

Re: speed up a logical replica setup

2024-03-07 Thread vignesh C
On Thu, 7 Mar 2024 at 10:05, Euler Taveira wrote: > > On Wed, Mar 6, 2024, at 7:02 AM, Hayato Kuroda (Fujitsu) wrote: > > Thanks for updating the patch! > > > Thanks for the feedback. I'm attaching v26 that addresses most of your > comments > and some issues pointed by Vignesh [1]. Few

Re: speed up a logical replica setup

2024-03-06 Thread Euler Taveira
On Wed, Mar 6, 2024, at 8:24 AM, vignesh C wrote: > Few comments: Thanks for your review. Some changes are included in v26. > 1) Can we use strdup here instead of atoi, as we do similarly in > case of pg_dump too, else we will do double conversion, convert using > atoi and again to string

Re: speed up a logical replica setup

2024-03-06 Thread Euler Taveira
On Wed, Mar 6, 2024, at 7:02 AM, Hayato Kuroda (Fujitsu) wrote: > Thanks for updating the patch! Thanks for the feedback. I'm attaching v26 that addresses most of your comments and some issues pointed by Vignesh [1]. > >* I also removed the check for standby is running. If the standby was >

Re: speed up a logical replica setup

2024-03-06 Thread vignesh C
On Sat, 2 Mar 2024 at 02:19, Euler Taveira wrote: > > On Thu, Feb 22, 2024, at 12:45 PM, Hayato Kuroda (Fujitsu) wrote: > > Based on idea from Euler, I roughly implemented. Thought? > > 0001-0013 were not changed from the previous version. > > V24-0014: addressed your comment in the replied

RE: speed up a logical replica setup

2024-03-06 Thread Hayato Kuroda (Fujitsu)
Dear Euler, Thanks for updating the patch! >v24-0003: as I said I don't think we need to add it, however, I won't fight >against it if people want to add this check. OK, let's wait comments from senior members. >Since I applied v24-0004, I realized that extra start / stop service are

Re: speed up a logical replica setup

2024-03-05 Thread Euler Taveira
On Tue, Mar 5, 2024, at 12:48 AM, Shubham Khanna wrote: > I applied the v25 patch and did RUN the 'pg_createsubscriber' command. > I was unable to execute it and experienced the following error: > > $ ./pg_createsubscriber -D node2/ -P "host=localhost port=5432 > dbname=postgres" -d postgres -d

Re: speed up a logical replica setup

2024-03-04 Thread Shlok Kyal
Hi, > I applied the v25 patch and did RUN the 'pg_createsubscriber' command. > I was unable to execute it and experienced the following error: > > $ ./pg_createsubscriber -D node2/ -P "host=localhost port=5432 > dbname=postgres" -d postgres -d db1 -p 9000 -r > ./pg_createsubscriber: invalid

Re: speed up a logical replica setup

2024-03-04 Thread Shubham Khanna
On Sat, Mar 2, 2024 at 2:19 AM Euler Taveira wrote: > > On Thu, Feb 22, 2024, at 12:45 PM, Hayato Kuroda (Fujitsu) wrote: > > Based on idea from Euler, I roughly implemented. Thought? > > 0001-0013 were not changed from the previous version. > > V24-0014: addressed your comment in the replied

Re: speed up a logical replica setup

2024-03-01 Thread Euler Taveira
ta copy and the replication progress tends to be faster on a physical replica. The purpose of this tool is to speed up a logical replica setup. --- doc/src/sgml/ref/allfiles.sgml|1 + doc/src/sgml/ref/pg_createsubscriber.sgml | 523 + doc/src/sgml/reference.sgml

Re: speed up a logical replica setup

2024-02-28 Thread Andrew Dunstan
On 2024-02-27 Tu 05:02, Hayato Kuroda (Fujitsu) wrote: Dear Euler, Sorry, which pg_rewind option did you mention? I cannot find. IIUC, -l is an only option which can accept the path, but it is not related with us. Sorry, I found [1]. I was confused with pg_resetwal. But my opinion was not

RE: speed up a logical replica setup

2024-02-27 Thread Hayato Kuroda (Fujitsu)
Dear Euler, > Sorry, which pg_rewind option did you mention? I cannot find. > IIUC, -l is an only option which can accept the path, but it is not related > with us. Sorry, I found [1]. I was confused with pg_resetwal. But my opinion was not so changed. The reason why --config-file exists is

RE: speed up a logical replica setup

2024-02-26 Thread Hayato Kuroda (Fujitsu)
Dear Euler, >> The possible solution would be >> 1) allow to run pg_createsubscriber if standby is initially stopped . >> I observed that pg_logical_createsubscriber also uses this approach. >> 2) read GUCs via SHOW command and restore them when server restarts >> >3. add a config-file option.

RE: speed up a logical replica setup

2024-02-25 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, I forgot to reply one of the suggestion. > 2) There is a CheckDataVersion function which does exactly this, will > we be able to use this: > + /* Check standby server version */ > + if ((ver_fd = fopen(versionfile, "r")) == NULL) > + pg_fatal("could not

Re: speed up a logical replica setup

2024-02-22 Thread Amit Kapila
On Fri, Feb 23, 2024 at 8:16 AM Euler Taveira wrote: > > On Wed, Feb 21, 2024, at 5:00 AM, Shlok Kyal wrote: > > I found some issues and fixed those issues with top up patches > v23-0012 and v23-0013 > 1. > Suppose there is a cascade physical replication node1->node2->node3. > Now if we run

Re: speed up a logical replica setup

2024-02-22 Thread Euler Taveira
On Wed, Feb 21, 2024, at 5:00 AM, Shlok Kyal wrote: > I found some issues and fixed those issues with top up patches > v23-0012 and v23-0013 > 1. > Suppose there is a cascade physical replication node1->node2->node3. > Now if we run pg_createsubscriber with node1 as primary and node2 as > standby,

Re: speed up a logical replica setup

2024-02-22 Thread Euler Taveira
On Thu, Feb 22, 2024, at 9:43 AM, Hayato Kuroda (Fujitsu) wrote: > > The possible solution would be > > 1) allow to run pg_createsubscriber if standby is initially stopped . > > I observed that pg_logical_createsubscriber also uses this approach. > > 2) read GUCs via SHOW command and restore them

Re: speed up a logical replica setup

2024-02-22 Thread 'Alvaro Herrera'
Hello, On 2024-Feb-22, Hayato Kuroda (Fujitsu) wrote: > Dear Alvaro, > > Hmm, but doesn't this mean that the server will log an ugly message > > that "client closed connection unexpectedly"? I think it's nicer to > > close the connection before terminating the process (especially > > since the

Re: speed up a logical replica setup

2024-02-22 Thread vignesh C
On Thu, 22 Feb 2024 at 21:17, Hayato Kuroda (Fujitsu) wrote: > > Dear Vignesh, > > > Few comments on the tests: > > 1) If the dry run was successful because of some issue then the server > > will be stopped so we can check for "pg_ctl status" if the server is > > running otherwise the connection

RE: speed up a logical replica setup

2024-02-22 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, > Few comments on the tests: > 1) If the dry run was successful because of some issue then the server > will be stopped so we can check for "pg_ctl status" if the server is > running otherwise the connection will fail in this case. Another way > would be to check if it does not have

RE: speed up a logical replica setup

2024-02-22 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, > Few comments: > 1) The below code can lead to assertion failure if the publisher is > stopped while dropping the replication slot: > + if (primary_slot_name != NULL) > + { > + conn = connect_database(dbinfo[0].pubconninfo); > + if (conn !=

RE: speed up a logical replica setup

2024-02-22 Thread Hayato Kuroda (Fujitsu)
Dear Vignesh, > Few comments regarding the documentation: > 1) max_replication_slots information seems to be present couple of times: > > + > + The target instance must have > + linkend="guc-max-replication-slots">max_replication_slots me> > + and

RE: speed up a logical replica setup

2024-02-22 Thread Hayato Kuroda (Fujitsu)
Dear Alvaro, > > 15. > > > > You said in case of failure, cleanups is not needed if the process exits > > soon [1]. > > But some functions call PQfinish() then exit(1) or pg_fatal(). Should we > > follow? > > Hmm, but doesn't this mean that the server will log an ugly message that > "client

RE: speed up a logical replica setup

2024-02-22 Thread Hayato Kuroda (Fujitsu)
Dear Shlok, > Hi, > > I have reviewed the v21 patch. And found an issue. > > Initially I started the standby server with a new postgresql.conf file > (not the default postgresql.conf that is present in the instance). > pg_ctl -D ../standby start -o "-c config_file=/new_path/postgresql.conf" >

  1   2   3   >