Re: Synchronizing slots from primary to standby

2023-10-31 Thread shveta malik
On Thu, Oct 26, 2023 at 5:43 PM Amit Kapila wrote: > > On Thu, Oct 26, 2023 at 5:38 PM Drouvot, Bertrand > wrote: > > > > On 10/26/23 10:40 AM, Amit Kapila wrote: > > > On Wed, Oct 25, 2023 at 8:49 PM Drouvot, Bertrand > > > wrote: > > >> > > > > > > Good point, I think we should enhance the

Re: Synchronizing slots from primary to standby

2023-10-27 Thread shveta malik
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wr

Re: Synchronizing slots from primary to standby

2023-10-27 Thread shveta malik
On Wed, Oct 25, 2023 at 3:15 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/25/23 5:00 AM, shveta malik wrote: > > On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 10/23/23 2:56 PM, shveta malik wr

Re: Synchronizing slots from primary to standby

2023-10-24 Thread shveta malik
On Tue, Oct 24, 2023 at 11:54 AM Drouvot, Bertrand wrote: > > Hi, > > On 10/23/23 2:56 PM, shveta malik wrote: > > On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand > > wrote: > > >> We are waiting for DEFAULT_NAPTIME_PER_CYCLE (3 minutes) before checking >

Re: Synchronizing slots from primary to standby

2023-10-24 Thread shveta malik
On Tue, Oct 24, 2023 at 3:35 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/24/23 7:44 AM, Ajin Cherian wrote: > > On Mon, Oct 23, 2023 at 11:22 PM Drouvot, Bertrand > > wrote: > >> > >> @@ -602,6 +602,9 @@ CreateDecodingContext(XLogRecPtr start_lsn, > >>

Re: Synchronizing slots from primary to standby

2023-10-23 Thread shveta malik
On Mon, Oct 23, 2023 at 5:52 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/20/23 5:27 AM, shveta malik wrote: > > On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > > > PFA v25 patch set. The changes are: > > > > 1) 'enable_failover' is changed to 'f

Re: Synchronizing slots from primary to standby

2023-10-19 Thread shveta malik
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > > > PFA v24 patch set which has below changes: > > > > 1) 'enable

Re: Synchronizing slots from primary to standby

2023-10-19 Thread shveta malik
On Wed, Oct 18, 2023 at 4:24 PM Amit Kapila wrote: > > On Tue, Oct 17, 2023 at 2:01 PM shveta malik wrote: > > > > On Tue, Oct 17, 2023 at 12:44 PM Peter Smith wrote: > > > > > > FYI - the latest patch failed to apply. > > > > > &g

Re: Synchronizing slots from primary to standby

2023-10-18 Thread shveta malik
On Wed, Oct 18, 2023 at 10:20 AM Amit Kapila wrote: > > On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand > wrote: > > > > On 10/13/23 10:35 AM, shveta malik wrote: > > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik > > > wrote: > > >> >

Re: Synchronizing slots from primary to standby

2023-10-17 Thread shveta malik
On Tue, Oct 17, 2023 at 9:06 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/13/23 10:35 AM, shveta malik wrote: > > On Thu, Oct 12, 2023 at 9:18 AM shveta malik wrote: > >> > > > > PFA v24 patch set which has below changes: > > > > 1) 'enable

Re: Synchronizing slots from primary to standby

2023-10-11 Thread shveta malik
On Tue, Oct 10, 2023 at 12:52 PM Peter Smith wrote: > > On Mon, Oct 9, 2023 at 9:34 PM shveta malik wrote: > > > > On Wed, Oct 4, 2023 at 8:53 AM Peter Smith wrote: > > > > > > Here are some review comments for v20-0002. > > > > > > >

Re: Synchronizing slots from primary to standby

2023-10-11 Thread shveta malik
On Tue, Oct 10, 2023 at 12:52 PM Peter Smith wrote: > > On Mon, Oct 9, 2023 at 9:34 PM shveta malik wrote: > > > > On Wed, Oct 4, 2023 at 8:53 AM Peter Smith wrote: > > > > > > Here are some review comments for v20-0002. > > > > > > >

Re: Synchronizing slots from primary to standby

2023-10-09 Thread shveta malik
On Mon, Oct 2, 2023 at 4:29 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shveta, > > Thank you for updating the patch! Thanks for the feedback Kuroda-san. I have addressed most of these in v22. Please find my comments inline. > > I found another ERROR due to the slot removal. Is this a real

Re: Synchronizing slots from primary to standby

2023-10-09 Thread shveta malik
On Mon, Oct 9, 2023 at 3:24 AM Peter Smith wrote: > > On Fri, Oct 6, 2023 at 7:37 PM Alvaro Herrera wrote: > > > > On 2023-Sep-27, Peter Smith wrote: > > > > > 3. get_local_synced_slot_names > > > > > > + for (int i = 0; i < max_replication_slots; i++) > > > + { > > > + ReplicationSlot *s =

Re: Synchronizing slots from primary to standby

2023-10-09 Thread shveta malik
On Wed, Oct 4, 2023 at 8:53 AM Peter Smith wrote: > > Here are some review comments for v20-0002. > Thanks Peter for the feedback. Comments from 31 till end are addressed in v22. First 30 comments will be addressed in the next version. > == > 1. GENERAL - errmsg/elog messages > > There are

Re: Synchronizing slots from primary to standby

2023-10-06 Thread shveta malik
On Fri, Oct 6, 2023 at 2:07 PM Alvaro Herrera wrote: > > On 2023-Sep-27, Peter Smith wrote: > > > 3. get_local_synced_slot_names > > > > + for (int i = 0; i < max_replication_slots; i++) > > + { > > + ReplicationSlot *s = >replication_slots[i]; > > + > > + /* Check if it is logical synchronized

Re: Synchronizing slots from primary to standby

2023-10-04 Thread shveta malik
On Wed, Oct 4, 2023 at 12:08 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/4/23 7:00 AM, shveta malik wrote: > > On Wed, Oct 4, 2023 at 9:56 AM shveta malik wrote: > > > The most simplistic approach would be: > > > > 1) maintain standby_sl

Re: Synchronizing slots from primary to standby

2023-10-04 Thread shveta malik
On Wed, Oct 4, 2023 at 5:00 PM Amit Kapila wrote: > > On Wed, Oct 4, 2023 at 11:55 AM Drouvot, Bertrand > wrote: > > > > On 10/4/23 6:26 AM, shveta malik wrote: > > > On Wed, Oct 4, 2023 at 5:36 AM Amit Kapila > > > wrote: > > >> > >

Re: Synchronizing slots from primary to standby

2023-10-03 Thread shveta malik
On Mon, Oct 2, 2023 at 4:29 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shveta, > > Thank you for updating the patch! > > I found another ERROR due to the slot removal. Is this a real issue? > > 1. applied add_sleep.txt, which emulated the case the tablesync worker stucked >and the primary

Re: Synchronizing slots from primary to standby

2023-10-03 Thread shveta malik
On Wed, Oct 4, 2023 at 9:56 AM shveta malik wrote: > > On Wed, Oct 4, 2023 at 5:36 AM Amit Kapila wrote: > > > > On Tue, Oct 3, 2023 at 9:27 PM shveta malik wrote: > > > > > > On Tue, Oct 3, 2023 at 7:56 PM Drouvot, Bertrand > > > wrote: > >

Re: Synchronizing slots from primary to standby

2023-10-03 Thread shveta malik
On Wed, Oct 4, 2023 at 5:36 AM Amit Kapila wrote: > > On Tue, Oct 3, 2023 at 9:27 PM shveta malik wrote: > > > > On Tue, Oct 3, 2023 at 7:56 PM Drouvot, Bertrand > > wrote: > > > > > > Hi, > > > > > > On 10/3/23 12:54 PM, Amit K

Re: Synchronizing slots from primary to standby

2023-10-03 Thread shveta malik
On Tue, Oct 3, 2023 at 7:56 PM Drouvot, Bertrand wrote: > > Hi, > > On 10/3/23 12:54 PM, Amit Kapila wrote: > > On Mon, Oct 2, 2023 at 11:39 AM Drouvot, Bertrand > > wrote: > >> > >> On 9/29/23 1:33 PM, Amit Kapila wrote: > >>> On Thu, Sep 28, 2023 at 6:31 PM Drouvot, Bertrand > >>> wrote: >

Re: Synchronizing slots from primary to standby

2023-09-27 Thread shveta malik
On Mon, Sep 25, 2023 at 7:46 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Ajin, Shveta, > > Thank you for rebasing the patch set! Here are new comments for v19_2-0001. > Thank You Kuroda-san for the feedback. Most of these are addressed in v20. Please find my response inline. > 01.

Re: Synchronizing slots from primary to standby

2023-09-24 Thread shveta malik
On Fri, Sep 22, 2023 at 3:48 PM Amit Kapila wrote: > > On Thu, Sep 21, 2023 at 9:16 AM shveta malik wrote: > > > > On Tue, Sep 19, 2023 at 10:29 AM shveta malik > > wrote: > > > > Currently in patch001, synchronize_slot_names is a GUC on both primary >

Re: Synchronizing slots from primary to standby

2023-09-24 Thread shveta malik
On Fri, Sep 22, 2023 at 6:01 PM Drouvot, Bertrand wrote: > > Hi, > > Thanks for all the work that has been done on this feature, and sorry > to have been quiet on it for so long. Thanks for looking into this. > > On 9/18/23 12:22 PM, shveta malik wrote: > > On W

Re: Synchronizing slots from primary to standby

2023-09-20 Thread shveta malik
On Tue, Sep 19, 2023 at 10:29 AM shveta malik wrote: > > On Fri, Sep 15, 2023 at 2:22 PM Peter Smith wrote: > > > > Hi. Here are some review comments for v17-0002. > > > > Thanks Peter for the feedback. I have addressed most of these in v18 > except 2. Ple

Re: Synchronizing slots from primary to standby

2023-09-18 Thread shveta malik
On Fri, Sep 15, 2023 at 2:22 PM Peter Smith wrote: > > Hi. Here are some review comments for v17-0002. > Thanks Peter for the feedback. I have addressed most of these in v18 except 2. Please find my comments for the ones not addressed. > This is a WIP and a long way from complete, but I

Re: Synchronizing slots from primary to standby

2023-09-18 Thread shveta malik
On Wed, Sep 13, 2023 at 4:48 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shveta, > > Sorry for the late response. > > > Thanks Kuroda-san for the feedback. > > > > > > 01. General > > > > > > I think the documentation can be added, not only GUCs. How about adding > > examples > > > for

Re: Synchronizing slots from primary to standby

2023-09-13 Thread shveta malik
On Wed, Sep 6, 2023 at 2:18 PM shveta malik wrote: > > On Fri, Sep 1, 2023 at 1:59 PM Peter Smith wrote: > > > > Hi Shveta. Here are some comments for patch v14-0002 > > > > The patch is large, so my code review is a WIP... more later next week... > > >

Re: Synchronizing slots from primary to standby

2023-09-10 Thread shveta malik
On Fri, Sep 8, 2023 at 4:40 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shveta, > > I resumed to check the thread. Here are my high-level comments. > Sorry if you have been already discussed. Thanks Kuroda-san for the feedback. > > 01. General > > I think the documentation can be added, not only

Re: Synchronizing slots from primary to standby

2023-09-06 Thread shveta malik
On Fri, Aug 25, 2023 at 2:15 PM Alvaro Herrera wrote: > > Wait a minute ... > > From bac0fbef8b203c530e5117b0b7cfee13cfab78b9 Mon Sep 17 00:00:00 2001 > From: Bharath Rupireddy > Date: Sat, 22 Jul 2023 10:17:48 + > Subject: [PATCH v13 1/2] Allow logical walsenders to wait for physical >

Re: Synchronizing slots from primary to standby

2023-09-06 Thread shveta malik
On Fri, Sep 1, 2023 at 1:59 PM Peter Smith wrote: > > Hi Shveta. Here are some comments for patch v14-0002 > > The patch is large, so my code review is a WIP... more later next week... > Thanks Peter for the feedback. I have tried to address most of these in v15. Please find my response inline

Re: Synchronizing slots from primary to standby

2023-08-29 Thread shveta malik
On Wed, Aug 23, 2023 at 4:21 PM Dilip Kumar wrote: > > On Wed, Aug 23, 2023 at 3:38 PM shveta malik wrote: > > > I have reviewed the v12-0002 patch and I have some comments. I see the > latest version posted sometime back and if any of this comment is > already fixed in t

Re: Synchronizing slots from primary to standby

2023-08-29 Thread shveta malik
On Fri, Aug 25, 2023 at 11:09 AM shveta malik wrote: > > On Wed, Aug 23, 2023 at 4:21 PM Dilip Kumar wrote: > > > > On Wed, Aug 23, 2023 at 3:38 PM shveta malik wrote: > > > > > I have reviewed the v12-0002 patch and I have some comments. I see the > &

Re: Synchronizing slots from primary to standby

2023-08-24 Thread shveta malik
On Wed, Aug 23, 2023 at 4:21 PM Dilip Kumar wrote: > > On Wed, Aug 23, 2023 at 3:38 PM shveta malik wrote: > > > I have reviewed the v12-0002 patch and I have some comments. I see the > latest version posted sometime back and if any of this comment is > already fixed in t

Re: Synchronizing slots from primary to standby

2023-08-23 Thread shveta malik
On Thu, Aug 17, 2023 at 4:09 PM shveta malik wrote: > > On Thu, Aug 17, 2023 at 11:55 AM shveta malik wrote: > > > > On Thu, Aug 17, 2023 at 11:44 AM Drouvot, Bertrand > > wrote: > > > > > > Hi, > > > > > > On 8/14/23 11:52 AM, s

Re: Synchronizing slots from primary to standby

2023-08-22 Thread shveta malik
On Thu, Aug 17, 2023 at 11:55 AM shveta malik wrote: > > On Thu, Aug 17, 2023 at 11:44 AM Drouvot, Bertrand > wrote: > > > > Hi, > > > > On 8/14/23 11:52 AM, shveta malik wrote: > > > > > > > > We (myself and Ajin) performed the te

Re: Adding a LogicalRepWorker type field

2023-08-17 Thread shveta malik
On Fri, Aug 18, 2023 at 8:50 AM Amit Kapila wrote: > > On Mon, Aug 14, 2023 at 12:08 PM Peter Smith wrote: > > > > The main patch for adding the worker type enum has been pushed [1]. > > > > Here is the remaining (rebased) patch for changing some previous > > cascading if/else to switch on the

Re: Synchronizing slots from primary to standby

2023-08-17 Thread shveta malik
On Thu, Aug 17, 2023 at 11:55 AM shveta malik wrote: > > On Thu, Aug 17, 2023 at 11:44 AM Drouvot, Bertrand > wrote: > > > > Hi, > > > > On 8/14/23 11:52 AM, shveta malik wrote: > > > > > > > > We (myself and Ajin) performed the te

Re: Synchronizing slots from primary to standby

2023-08-17 Thread shveta malik
On Thu, Aug 17, 2023 at 11:44 AM Drouvot, Bertrand wrote: > > Hi, > > On 8/14/23 11:52 AM, shveta malik wrote: > > > > > We (myself and Ajin) performed the tests to compute the lag in standby > > slots as compared to primary slots with different number of

Re: Synchronizing slots from primary to standby

2023-08-14 Thread shveta malik
On Mon, Aug 14, 2023 at 3:22 PM shveta malik wrote: > > On Tue, Aug 8, 2023 at 11:11 AM Drouvot, Bertrand > wrote: > > > > Hi, > > > > On 8/8/23 7:01 AM, shveta malik wrote: > > > On Mon, Aug 7, 2023 at 3:17 PM Drouvot, Bertrand > > > wrote

Re: Synchronizing slots from primary to standby

2023-08-14 Thread shveta malik
On Tue, Aug 8, 2023 at 11:11 AM Drouvot, Bertrand wrote: > > Hi, > > On 8/8/23 7:01 AM, shveta malik wrote: > > On Mon, Aug 7, 2023 at 3:17 PM Drouvot, Bertrand > > wrote: > >> > >> Hi, > >> > >> On 8/4/23 1:32 PM, shveta malik w

Re: Synchronizing slots from primary to standby

2023-08-07 Thread shveta malik
On Mon, Aug 7, 2023 at 3:17 PM Drouvot, Bertrand wrote: > > Hi, > > On 8/4/23 1:32 PM, shveta malik wrote: > > On Fri, Aug 4, 2023 at 2:44 PM Drouvot, Bertrand > > wrote: > >> On 7/28/23 4:39 PM, Bharath Rupireddy wrote: > > >> Sorry to be late, b

Re: Synchronizing slots from primary to standby

2023-08-07 Thread shveta malik
On Tue, Aug 1, 2023 at 4:52 PM shveta malik wrote: > > On Thu, Jul 27, 2023 at 12:13 PM shveta malik wrote: > > > > On Thu, Jul 27, 2023 at 10:55 AM Amit Kapila > > wrote: > > > > > > On Wed, Jul 26, 2023 at 10:31 AM shveta malik > > > wro

Re: Synchronizing slots from primary to standby

2023-08-04 Thread shveta malik
On Fri, Aug 4, 2023 at 2:44 PM Drouvot, Bertrand wrote: > > Hi, > > On 7/28/23 4:39 PM, Bharath Rupireddy wrote: > > On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila wrote: > >> > >>> 2. All candidate standbys will start one slot sync worker per logical > >>> slot which might not be scalable. > >> >

Re: Synchronizing slots from primary to standby

2023-08-03 Thread shveta malik
On Thu, Aug 3, 2023 at 12:28 AM Bharath Rupireddy wrote: > > On Tue, Aug 1, 2023 at 5:01 PM shveta malik wrote: > > > > > The work division amongst the sync workers can > > > be simple, the logical replication launcher builds a shared memory > > > str

Re: Synchronizing slots from primary to standby

2023-08-01 Thread shveta malik
On Fri, Jul 28, 2023 at 8:54 PM Bharath Rupireddy wrote: > > On Thu, Jul 27, 2023 at 10:55 AM Amit Kapila wrote: > > > > I wonder if we anyway some sort of design like this because we > > shouldn't allow to spawn as many workers as the number of databases. > > There has to be some existing or

Re: Synchronizing slots from primary to standby

2023-07-27 Thread shveta malik
On Thu, Jul 27, 2023 at 10:55 AM Amit Kapila wrote: > > On Wed, Jul 26, 2023 at 10:31 AM shveta malik wrote: > > > > On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila wrote: > > > > > > On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy > > > wrote:

Re: Synchronizing slots from primary to standby

2023-07-26 Thread shveta malik
On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy wrote: > > On Fri, Jul 21, 2023 at 5:16 PM shveta malik wrote: > > > > Thanks Bharat for letting us know. It is okay to split the patch, it > > may definitely help to understand the modules better but shall we take

Re: Synchronizing slots from primary to standby

2023-07-25 Thread shveta malik
On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila wrote: > > On Mon, Jul 24, 2023 at 8:03 AM Bharath Rupireddy > wrote: > > > > On Fri, Jul 21, 2023 at 5:16 PM shveta malik wrote: > > > > > > Thanks Bharat for letting us know. It is okay to split the patch, it

Re: Synchronizing slots from primary to standby

2023-07-21 Thread shveta malik
On Fri, Jul 21, 2023 at 11:36 AM Bharath Rupireddy wrote: > > On Thu, Jul 20, 2023 at 5:05 PM shveta malik wrote: > > > > On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila wrote: > > > > > > On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand > > &

Re: Synchronizing slots from primary to standby

2023-07-20 Thread shveta malik
On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila wrote: > > On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand > wrote: > > > 3. As mentioned in the initial email, I think it would be better to > replace LIST_SLOTS command with a SELECT query. > I had a look at this thread. I am interested to work

Re: Support logical replication of DDLs

2023-06-22 Thread shveta malik
On Wed, Jun 21, 2023 at 6:38 PM Jelte Fennema wrote: > > (to be clear I only skimmed the end of this thread and did not look at > all the previous messages) > > I took a quick look at the first patch (about deparsing table ddl) and > it seems like this would also be very useful for a SHOW CREATE

Re: Support logical replication of DDLs

2023-06-21 Thread shveta malik
On Mon, Jun 12, 2023 at 7:17 AM Wei Wang (Fujitsu) wrote: > > On Thur, Jun 8, 2023 20:02 PM shveta malik wrote: > > Thank You Vignesh for handling (a), Ajin for handling (b), Shi-san and > > Hou-san for contributing in (c). > > > > The new changes are in p

Re: Support logical replication of DDLs

2023-06-18 Thread shveta malik
As per suggestion by Amit, reviewed two more formats to be used for DDL's WAL-logging purpose, analysis below: NodeToString: I do not think it is a good idea to use NodeToString in DDL Rep for reasons below: 1) It consists of too much internal and not-needed information. 2) Too large to be logged

Re: Support logical replication of DDLs

2023-06-08 Thread shveta malik
On Thu, Jun 8, 2023 at 5:31 PM shveta malik wrote: > > Please find new set of patches addressing below: > a) Issue mentioned by Wang-san in [1], > b) Comments from Peter given in [2] > c) Comments from Amit given in the last 2 emails. > > [1]: > https://www.p

Re: Support logical replication of DDLs

2023-06-08 Thread shveta malik
On Tue, Jun 6, 2023 at 11:31 AM Wei Wang (Fujitsu) wrote: > > On Thur, June 1, 2023 at 23:42 vignesh C wrote: > > On Wed, 31 May 2023 at 14:32, Wei Wang (Fujitsu) > > wrote: > > > ~~~ > > > > > > 2. Deparsed results of the partition table. > > > When I run the following SQLs: > > > ``` > > >

Re: Support logical replication of DDLs

2023-06-08 Thread shveta malik
On Tue, Jun 6, 2023 at 4:26 PM Amit Kapila wrote: > > On Mon, Jun 5, 2023 at 3:00 PM shveta malik wrote: > > > > Few assorted comments: Hi Amit, thanks for the feedback. Addressed these in recent patch posted (*2023_06_08.patch) > === > 1. I see th

Re: Support logical replication of DDLs

2023-05-29 Thread shveta malik
On Mon, May 29, 2023 at 6:16 PM vignesh C wrote: > > > > > I found few comments while making some changes to the patch: > > 1) Now that objtree is removed, these comments should be modified: > > * Deparse object tree is created by using: > > * a) new_objtree("know contents") where the complete

pg_get_indexdef() modification to use TxnSnapshot

2023-05-26 Thread shveta malik
I have attempted to convert pg_get_indexdef() to use systable_beginscan() based on transaction-snapshot rather than using SearchSysCache(). The latter does not have any old info and thus provides only the latest info as per the committed txns, which could result in errors in some scenarios. One

Re: Support logical replication of DDLs

2023-05-08 Thread shveta malik
On Mon, May 8, 2023 at 3:58 PM shveta malik wrote: > > On Tue, May 2, 2023 at 8:30 AM shveta malik wrote: > > > > On Fri, Apr 28, 2023 at 5:11 PM Amit Kapila wrote: > > > > > > Now, I think we can try to eliminate this entire ObjTree machinery and >

Re: Support logical replication of DDLs

2023-05-01 Thread shveta malik
On Fri, Apr 28, 2023 at 5:11 PM Amit Kapila wrote: > > On Tue, Apr 25, 2023 at 9:28 AM Zhijie Hou (Fujitsu) > wrote: > > > > I have a few high-level comments on the deparsing approach used in the > patch. As per my understanding, we first build an ObjTree from the DDL > command, then convert the

Re: Support logical replication of DDLs

2023-04-23 Thread shveta malik
On Thu, Apr 20, 2023 at 3:40 PM Amit Kapila wrote: > > On Thu, Apr 20, 2023 at 9:11 AM shveta malik wrote: > > > > > > Few comments for ddl_deparse.c in patch dated April17: > > > > > 6) There are plenty of places where we use 'append_not_present' > &g

Re: Support logical replication of DDLs

2023-04-20 Thread shveta malik
On Thu, Apr 20, 2023 at 2:28 PM shveta malik wrote: > > On Thu, Apr 20, 2023 at 9:11 AM shveta malik wrote: >> >> On Mon, Apr 17, 2023 at 5:32 PM Zhijie Hou (Fujitsu) >> wrote: >> > >> > Attach the new version patch set which include the following

Re: Support logical replication of DDLs

2023-04-20 Thread shveta malik
On Thu, Apr 20, 2023 at 9:11 AM shveta malik wrote: > On Mon, Apr 17, 2023 at 5:32 PM Zhijie Hou (Fujitsu) > wrote: > > > > Attach the new version patch set which include the following changes: > > > > Few comments for ddl_deparse.c in patch dated April17: &g

Re: Support logical replication of DDLs

2023-04-19 Thread shveta malik
On Mon, Apr 17, 2023 at 5:32 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, April 10, 2023 7:20 PM Amit Kapila wrote: > > > > On Fri, Apr 7, 2023 at 8:52 AM houzj.f...@fujitsu.com > > wrote: > > > > > > Sorry, there was a miss when rebasing the patch which could cause the > > > CFbot to fail and

Re: Support logical replication of DDLs

2023-04-04 Thread shveta malik
On Tue, Apr 4, 2023 at 8:43 AM houzj.f...@fujitsu.com wrote: > Attach the new version patch set which did the following changes: > Hello, I tried below: pubnew=# ALTER PUBLICATION mypub2 SET (ddl = 'table'); ALTER PUBLICATION pubnew=# \dRp+ Publication

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-08 Thread shveta malik
On Tue, Feb 7, 2023 at 8:18 AM shiy.f...@fujitsu.com wrote: > > > On Thu, Feb 2, 2023 11:48 AM shveta malik wrote: > > > > > > So to fix this, I think either we update origin and slot entries in > > the system catalog after the creation has passed or we clean-

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-03 Thread shveta malik
On Thu, Feb 2, 2023 at 5:01 PM shveta malik wrote: > > Reviewing further > Hi Melih, int64 rep_slot_id; int64 lastusedid; int64 sublastusedid --Should all of the above be unsigned integers? thanks Shveta

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-02 Thread shveta malik
On Fri, Feb 3, 2023 at 11:50 AM shveta malik wrote: > > On Thu, Feb 2, 2023 at 5:01 PM shveta malik wrote: > > > > 2e) > When it goes to reuse flow (i.e. before walrcv_slot_snapshot), if > needed we can dump newly obtained origin_startpos also: > > erepor

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-02 Thread shveta malik
On Thu, Feb 2, 2023 at 5:01 PM shveta malik wrote: > > Reviewing further > Few more comments for v10-0002 and v7-0001: 1) + * need_full_snapshot + * if true, create a snapshot able to read all tables, + * otherwise do not create any snapshot. + * CreateDecodi

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-02 Thread shveta malik
On Wed, Feb 1, 2023 at 5:37 PM Melih Mutlu wrote: > > Hi, > Please see attached patches. > > Thanks, > -- > Melih Mutlu > Microsoft Hi Melih, Few suggestions on v10-0002-Reuse patch 1) for (int64 i = 1; i <= lastusedid; i++) { char

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-02 Thread shveta malik
On Thu, Feb 2, 2023 at 9:18 AM shveta malik wrote: > > > Hi Melih, > I think I am able to identify the root cause. It is not memory > corruption, but the way origin-names are stored in system-catalog > mapped to a particular relation-id before even those are created. > A

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-01 Thread shveta malik
On Wed, Feb 1, 2023 at 5:42 PM Melih Mutlu wrote: > > Hi Shveta, > > shveta malik , 1 Şub 2023 Çar, 15:01 tarihinde şunu > yazdı: >> >> On Wed, Feb 1, 2023 at 5:05 PM Melih Mutlu wrote: >> 2) I found a crash in the previous patch (v9), but have not tested it &g

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-01 Thread shveta malik
On Wed, Feb 1, 2023 at 5:42 PM Melih Mutlu wrote: > > > Thanks for investigating this error. I think it's the same error as the one > Shi reported earlier. [1] > I couldn't reproduce it yet but will apply your tweaks and try again. > Looking into this. > > [1] >

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-02-01 Thread shveta malik
On Wed, Feb 1, 2023 at 5:05 PM Melih Mutlu wrote: > > Hi, > > Please see attached patches for the below changes. > > Thanks for reviewing, > -- > Melih Mutlu > Microsoft Hello Melih, Thank you for making the changes. I have few more comments: 1) src/backend/replication/logical/worker.c:

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-01-31 Thread shveta malik
On Tue, Jan 31, 2023 at 3:57 PM wangw.f...@fujitsu.com wrote: > > On Mon, Jan 23, 2023 21:00 PM Melih Mutlu wrote: > > Hi, > > > > Thanks for your review. > > Attached updated versions of the patches. > > Thanks for updating the patch set. > > > > 5. New member "created_slot" in structure

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-01-31 Thread shveta malik
On Fri, Jan 27, 2023 at 3:41 PM shveta malik wrote: > > > I am reviewing further... > thanks > Shveta Few more comments: v4-0001: 1) REPLICATION_SLOT_SNAPSHOT --Do we need 'CREATE' prefix with it i.e. CREATE_REPLICATION_SNAPSHOT (or some other brief

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-01-27 Thread shveta malik
On Thu, Jan 26, 2023 at 7:53 PM Melih Mutlu wrote: > > If a relation is currently being synced by a tablesync worker and uses a > replication slot/origin for that operation, then srrelslotname and > srreloriginname fields will have values. > When a relation is done with its replication

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-01-25 Thread shveta malik
On Mon, Jan 23, 2023 at 6:30 PM Melih Mutlu wrote: > > Hi, > > Thanks for your review. > Attached updated versions of the patches. > Hello, I am still in the process of reviewing the patch, before that I tried to run below test: --publisher create table tab1(id int , name varchar); create table

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-24 Thread shveta malik
On Tue, Jan 24, 2023 at 5:49 PM Takamichi Osumi (Fujitsu) wrote: > > > Attached the patch v20 that has incorporated all comments so far. > Kindly have a look at the attached patch. > > > Best Regards, > Takamichi Osumi > Thank You for patch. My previous comments are addressed. Tested it

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-20 Thread shveta malik
On Fri, Jan 20, 2023 at 2:23 PM shveta malik wrote: > > On Fri, Jan 20, 2023 at 1:08 PM Peter Smith wrote: > > > a) the message should say that this is the *remaining* time to left to wait. > > > > b) it might be convenient to know from the log what was the origina

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-20 Thread shveta malik
On Fri, Jan 20, 2023 at 1:08 PM Peter Smith wrote: > a) the message should say that this is the *remaining* time to left to wait. > > b) it might be convenient to know from the log what was the original > min_apply_delay value in the 1st place. > > For example, the logs might look something like

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-19 Thread shveta malik
On Thu, Jan 19, 2023 at 12:42 PM Takamichi Osumi (Fujitsu) wrote: > > On Wednesday, January 18, 2023 4:06 PM Peter Smith > wrote: > > Here are my review comments for the latest patch v16-0001. (excluding the > > test code) > Hi, thank you for your review ! > > > == > > > > General > > > >

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-19 Thread shveta malik
On Thu, Jan 19, 2023 at 3:44 PM shveta malik wrote: > > On Thu, Jan 19, 2023 at 11:11 AM Amit Kapila wrote: > > > > On Wed, Jan 18, 2023 at 12:09 PM Amit Kapila > > wrote: > > > > > > On Fri, Jan 13, 2023 at 11:50 AM Peter Smith > > > w

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-19 Thread shveta malik
On Thu, Jan 19, 2023 at 11:11 AM Amit Kapila wrote: > > On Wed, Jan 18, 2023 at 12:09 PM Amit Kapila wrote: > > > > On Fri, Jan 13, 2023 at 11:50 AM Peter Smith wrote: > > > > > > Here are some review comments for patch v79-0002. > > > > > > > So, this is about the latest

Re: Question about initial logical decoding snapshot

2023-01-18 Thread shveta malik
Hello, I was curious as to why we need 3rd running_xact and wanted to learn more about it, so I have made a few changes to come up with a patch which builds the snapshot in 2 running_xacts. The motive is to run the tests to see the failures/issues with this approach to understand the need of

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-16 Thread shveta malik
On Tue, Jan 17, 2023 at 9:07 AM houzj.f...@fujitsu.com wrote: > > On Tuesday, January 17, 2023 11:32 AM Peter Smith > wrote: > > > > On Tue, Jan 17, 2023 at 1:21 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Tuesday, January 17, 2023 5:43 AM Peter Smith > > wrote: > > > > > > > > On

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-12 Thread shveta malik
On Thu, Jan 12, 2023 at 4:37 PM Amit Kapila wrote: > > > But then do you suggest that tomorrow if we allow parallel sync > workers then we have a separate column leader_sync_pid? I think that > doesn't sound like a good idea and moreover one can refer to docs for > clarification. > > -- okay,

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-12 Thread shveta malik
On Thu, Jan 12, 2023 at 10:34 AM Amit Kapila wrote: > > On Thu, Jan 12, 2023 at 9:54 AM Peter Smith wrote: > > > > > > doc/src/sgml/monitoring.sgml > > > > 5. pg_stat_subscription > > > > @@ -3198,11 +3198,22 @@ SELECT pid, wait_event_type, wait_event FROM > > pg_stat_activity WHERE wait_event i

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-12 Thread shveta malik
On Wed, Jan 11, 2023 at 6:16 PM Hayato Kuroda (Fujitsu) wrote: > > > 2. > > I think users can set ' wal_receiver_status_interval ' to 0 or more > > than 'wal_sender_timeout'. But is this a frequent use-case scenario or > > do we see DBAs setting these in such a way by mistake? If so, then I > >

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-11 Thread shveta malik
On Wed, Jan 11, 2023 at 6:16 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shveta, > > Thanks for reviewing! PSA new version. > > > 1. > > + errmsg("min_apply_delay must not be set when streaming = parallel"))); > > we give the same error msg for both the cases: > > a. when subscription is created

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-11 Thread shveta malik
On Wed, Jan 11, 2023 at 3:27 PM shveta malik wrote: > > On Tue, Jan 10, 2023 at 7:42 PM Takamichi Osumi (Fujitsu) > wrote: > > > > On Tuesday, January 3, 2023 4:01 PM vignesh C wrote: > > Hi, thanks for your review ! > > > > > > > 1) This g

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-11 Thread shveta malik
On Tue, Jan 10, 2023 at 7:42 PM Takamichi Osumi (Fujitsu) wrote: > > On Tuesday, January 3, 2023 4:01 PM vignesh C wrote: > Hi, thanks for your review ! > > > > 1) This global variable can be removed as it is used only in send_feedback > > which > > is called from maybe_delay_apply so we could

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-03 Thread shveta malik
> > On Tue, 27 Dec 2022 at 14:59, Hayato Kuroda (Fujitsu) > wrote: > > Note that more than half of the modifications are done by Osumi-san. > Please find a few minor comments. 1. + diffms = TimestampDifferenceMilliseconds(GetCurrentTimestamp(), + TimestampTzPlusMilliseconds(ts,

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-03 Thread shveta malik
On Tue, Jan 3, 2023 at 11:10 AM wangw.f...@fujitsu.com wrote: > > On Mon, Jan 2, 2023 at 18:54 PM Amit Kapila wrote: > > On Fri, Dec 30, 2022 at 3:55 PM wangw.f...@fujitsu.com > > wrote: > > > > > > I've checked it and it looks good to me. > > > Rebased the other patches and ran the pgident for

Re: Force streaming every change in logical decoding

2022-12-23 Thread shveta malik
On Fri, Dec 23, 2022 at 2:03 PM shiy.f...@fujitsu.com wrote: > > On Fri, Dec 23, 2022 1:50 PM Amit Kapila > > > > On Thu, Dec 22, 2022 at 6:18 PM shiy.f...@fujitsu.com > > wrote: > > > > > > > > > Besides, I tried to reduce data size in streaming subscription tap tests by this > > > new GUC

Re: Force streaming every change in logical decoding

2022-12-20 Thread shveta malik
Going with ' logical_decoding_work_mem' seems a reasonable solution, but since we are mixing the functionality of developer and production GUC, there is a slight risk that customer/DBAs may end up setting it to 0 and forget about it and thus hampering system's performance. Have seen many such

<    1   2   3