Re: logical decoding and replication of sequences, take 2

2023-07-11 Thread Amit Kapila
all WAL records that the DDL execution generates. > Yes, the DDL replication patch uses the parse tree and catalog information to generate a deparsed form of DDL statement which is WAL logged and used to replicate DDLs. -- With Regards, Amit Kapila.

Re: doc: clarify the limitation for logical replication when REPILICA IDENTITY is FULL

2023-07-11 Thread Amit Kapila
sounds like we are checking whether datatype has any operator class for btree or hash access methods but we are actually checking if there is an equality operator (function) defined in the default op class for those access methods. Am, I missing something? -- With Regards, Amit Kapila.

Re: doc: clarify the limitation for logical replication when REPILICA IDENTITY is FULL

2023-07-10 Thread Amit Kapila
r seeing this, I am thinking about whether we add this restriction on the Subscription page [1] or Restrictions page [2] as proposed. Do you others have any preference? [1] - https://www.postgresql.org/docs/devel/logical-replication-subscription.html [2] - https://www.postgresql.org/docs/devel/logical-replication-restrictions.html -- With Regards, Amit Kapila.

Re: doc: improve the restriction description of using indexes on REPLICA IDENTITY FULL table.

2023-07-10 Thread Amit Kapila
On Tue, Jul 11, 2023 at 4:54 AM Peter Smith wrote: > > On Mon, Jul 10, 2023 at 2:21 PM Amit Kapila wrote: > > > > On Mon, Jul 10, 2023 at 7:55 AM Peter Smith wrote: > > > > > > On Sat, Jul 8, 2023 at 1:49 PM Amit Kapila > > > wrote: > >

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-10 Thread Amit Kapila
03 basically improved performance from first two patches > > Agree, 0003 is definitely a good addition which was missing earlier. > +1. [1] - https://www.postgresql.org/message-id/CAA4eK1JOZHmy2o2F2wTCPKsjpwDiKZPOeTa_jt%3Dwm2JLbf-jsg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: [Patch] Use *other* indexes on the subscriber when REPLICA IDENTITY is FULL

2023-07-10 Thread Amit Kapila
brought up the amgettuple issue, which is fair. So, > overall, as far as I can see, we can > easily add hash indexes but all others are either very hard to add or not > possible. > Agreed. So, let's update the patch with comments indicating the challenges for supporting the other index types than btree and hash. -- With Regards, Amit Kapila.

Re: doc: clarify the limitation for logical replication when REPILICA IDENTITY is FULL

2023-07-10 Thread Amit Kapila
r example, debezium. > Can you explain the difference and problem you are seeing? As per my understanding, this is the behavior from the time logical replication has been introduced. [1] - https://www.postgresql.org/message-id/TYAPR01MB5866C7B6086EB74918910F74F527A%40TYAPR01MB5866.jpnprd01.prod.outlook.com -- With Regards, Amit Kapila.

Re: doc: clarify the limitation for logical replication when REPILICA IDENTITY is FULL

2023-07-10 Thread Amit Kapila
nnot be replicated for the published tables that specifies REPLICA IDENTITY FULL but the table includes an attribute whose datatype doesn't have an equality operator defined for it on the subscriber."? -- With Regards, Amit Kapila.

Re: [Patch] Use *other* indexes on the subscriber when REPLICA IDENTITY is FULL

2023-07-10 Thread Amit Kapila
? As per my understanding if we don't have PK or replica identity then after the index scan, we do tuples_equal which will fail for GIST or SP-GIST. Am, I missing something? -- With Regards, Amit Kapila.

Re: doc: improve the restriction description of using indexes on REPLICA IDENTITY FULL table.

2023-07-09 Thread Amit Kapila
On Mon, Jul 10, 2023 at 7:55 AM Peter Smith wrote: > > On Sat, Jul 8, 2023 at 1:49 PM Amit Kapila wrote: > > > > On Fri, Jul 7, 2023 at 1:36 PM Masahiko Sawada > > wrote: > > > > > > I prefer the first suggestion. I've attached the updated patch.

Re: Synchronizing slots from primary to standby

2023-07-09 Thread Amit Kapila
On Sun, Jul 9, 2023 at 1:01 PM Bharath Rupireddy wrote: > > On Fri, Jun 16, 2023 at 3:26 PM Amit Kapila wrote: > > > > > > 1. Can you please try to explain the functionality of the overall > > patch somewhere in the form of comments and or commit message? &g

Re: Initial Schema Sync for Logical Replication

2023-07-08 Thread Amit Kapila
the individual tablesync workers can sync the schema of partitions and their data. And when publish_via_root is true, then the table sync worker is responsible to sync parent and child tables along with data. Do you think such a mechanism can address the partition table related cases? -- With Regards, Amit Kapila.

Re: [Patch] Use *other* indexes on the subscriber when REPLICA IDENTITY is FULL

2023-07-07 Thread Amit Kapila
iction. It has existed before > the > commit. I (or my colleague) will post the patch to add the description, maybe > after [1] is committed. > I don't think there is any dependency between the two. So, feel free to post the patch separately. -- With Regards, Amit Kapila.

Re: doc: improve the restriction description of using indexes on REPLICA IDENTITY FULL table.

2023-07-07 Thread Amit Kapila
how about: "Candidate indexes must be btree, non-partial, and the leftmost index column must be a non-expression and reference to a published table column (i.e. cannot consist of only expressions)."? -- With Regards, Amit Kapila.

Re: logicalrep_message_type throws an error

2023-07-06 Thread Amit Kapila
On Wed, Jul 5, 2023 at 7:54 PM Ashutosh Bapat wrote: > > On Wed, Jul 5, 2023 at 7:07 PM Euler Taveira wrote: > > > > On Wed, Jul 5, 2023, at 7:56 AM, Alvaro Herrera wrote: > > > > On 2023-Jul-05, Amit Kapila wrote: > > > > > I think after

Re: logicalrep_message_type throws an error

2023-07-06 Thread Amit Kapila
no invalid message type will be reported. > Yeah, but it would still be displayed both in context and the actual message. > I guess there's a > point to that argument as well. > One point to note is that the user may also get confused if the actual ERROR says message type as 'X' and the context says '???'. I feel in this case duplicate information is better than different information. -- With Regards, Amit Kapila.

Re: pg_decode_message vs skip_empty_xacts and xact_wrote_changes

2023-07-06 Thread Amit Kapila
On Wed, Jul 5, 2023 at 7:20 PM Ashutosh Bapat wrote: > > On Wed, Jul 5, 2023 at 2:29 PM Amit Kapila wrote: > > > > On Wed, Jul 5, 2023 at 2:28 PM Amit Kapila wrote: > > > > > > On Mon, Jul 3, 2023 at 4:49 PM vignesh C wrote: > > > > > > >

Re: Assert while autovacuum was executing

2023-07-05 Thread Amit Kapila
On Fri, Jun 30, 2023 at 8:26 AM Amit Kapila wrote: > > On Wed, Jun 28, 2023 at 7:26 AM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks for the verification. Unless someone has any further comments > or suggestions, I'll push this next week sometime. > Pushed but forg

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-05 Thread Amit Kapila
mount of > improvement, but not sure about the later patches yet. > I suggest then we should focus first on those 3, get them committed and then look at the remaining. > Amit Kapila , 3 Tem 2023 Pzt, 08:59 tarihinde > şunu yazdı: > > On thinking about this, I think the primary benefit we

Re: logicalrep_message_type throws an error

2023-07-05 Thread Amit Kapila
On Wed, Jul 5, 2023 at 4:26 PM Alvaro Herrera wrote: > > On 2023-Jul-05, Amit Kapila wrote: > > > I think after returning "???" from logicalrep_message_type(), the > > above is possible when we get the error: "invalid logical replication > > message typ

Re: doc: improve the restriction description of using indexes on REPLICA IDENTITY FULL table.

2023-07-05 Thread Amit Kapila
On Wed, Jul 5, 2023 at 12:02 PM Masahiko Sawada wrote: > > On Wed, Jul 5, 2023 at 2:46 PM Amit Kapila wrote: > > > > On Wed, Jul 5, 2023 at 9:01 AM Peter Smith wrote: > > > > > > Hi. Here are some review comments for this patc

Re: pg_decode_message vs skip_empty_xacts and xact_wrote_changes

2023-07-05 Thread Amit Kapila
On Wed, Jul 5, 2023 at 2:28 PM Amit Kapila wrote: > > On Mon, Jul 3, 2023 at 4:49 PM vignesh C wrote: > > > > +1 for the first version patch, I also felt the first version is > > easily understandable. > > > > Okay, please find the slightly updated version (ch

Re: pg_decode_message vs skip_empty_xacts and xact_wrote_changes

2023-07-05 Thread Amit Kapila
a day or two. -- With Regards, Amit Kapila.

Re: doc: improve the restriction description of using indexes on REPLICA IDENTITY FULL table.

2023-07-04 Thread Amit Kapila
t of these checks indicates that the leftmost column of the index should be non-expression, second and third indicates what you suggest in your wording. We can also think that what you wrote in a way is a superset of "leftmost index column is a non-expression" and "leftmost index column should be present in remote rel" but I feel it would be better to explicit about the first part as it is easy to understand for users at least in docs. -- With Regards, Amit Kapila.

Re: logicalrep_message_type throws an error

2023-07-04 Thread Amit Kapila
n what about the case when we forgot to handle some message in logicalrep_message_type() but handled it in apply_dispatch()? Isn't it better to return the 'action' from the function logicalrep_message_type() for unknown type? That way the information could be a bit better and we can easily catch the code bug as well. -- With Regards, Amit Kapila.

Re: doc: improve the restriction description of using indexes on REPLICA IDENTITY FULL table.

2023-07-04 Thread Amit Kapila
Yeah, I think it is good to mention that. Accordingly, the comments atop build_replindex_scan_key(), FindUsableIndexForReplicaIdentityFull(), IsIndexOnlyOnExpression() should also be adjusted. -- With Regards, Amit Kapila.

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-02 Thread Amit Kapila
On Mon, Jul 3, 2023 at 9:42 AM Amit Kapila wrote: > > On Wed, Jun 28, 2023 at 12:02 PM Hayato Kuroda (Fujitsu) > wrote: > > > But in the later patch the tablesync worker tries to reuse the slot during > > the > > synchronization, so in this case the a

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-07-02 Thread Amit Kapila
drop them but at this stage I feel we need to investigate why those are not helping? -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-06-30 Thread Amit Kapila
lots on primary, so launcher disconnects from primary and waits a time > period (up to 3min). > Even if the apply worker creates the slot on publisher, but the launcher on > standby > cannot notice that. The synchronization may start 3 min later. > I feel this should be based on some GUC l

Re: pg_decode_message vs skip_empty_xacts and xact_wrote_changes

2023-06-29 Thread Amit Kapila
eceived for this transaction during which we would have outputted > the begin message, this condition is to skip outputting begin message > if the begin message was already outputted. > I feel the use of last_write has reduced the readability of this part of the code. It may be that we can add comments to make it clear but I feel your previous version was much easier to understand. -- With Regards, Amit Kapila.

Re: pg_decode_message vs skip_empty_xacts and xact_wrote_changes

2023-06-29 Thread Amit Kapila
--- BEGIN 739 message: transactional: 1 prefix: test, sz: 4 content:msg1 COMMIT 739 message: transactional: 0 prefix: test, sz: 4 content:msg2 (4 rows) Thoughts? -- With Regards, Amit Kapila.

Re: Assert while autovacuum was executing

2023-06-29 Thread Amit Kapila
On Wed, Jun 28, 2023 at 7:26 AM Zhijie Hou (Fujitsu) wrote: > > On Monday, June 26, 2023 12:18 PM Amit Kapila wrote: > > > > Fair enough. If we could have been convinced of this then we can keep the > > required change only for HEAD. But anyway let's remove the work rel

Re: Synchronizing slots from primary to standby

2023-06-29 Thread Amit Kapila
ync slot workers corresponding to slots that need sync on standby and apply workers for active subscriptions on primary (which will be a subscriber in this context). If this is correct, then do you expect to launch a separate kind of standby launcher for sync slots? -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-06-29 Thread Amit Kapila
On Wed, Jun 28, 2023 at 12:19 PM Drouvot, Bertrand wrote: > > On 6/26/23 12:34 PM, Amit Kapila wrote: > > On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand > > wrote: > >> > >> On 6/20/23 12:22 PM, Amit Kapila wrote: > >>> On Mon, Jun 19

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-06-28 Thread Amit Kapila
_flush in memory but not in the disk? BTW, have we ensured that discarding the additional records are already sent to the subscriber, if so, why for those records confirmed_flush LSN is not progressed? -- With Regards, Amit Kapila.

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-06-27 Thread Amit Kapila
0003) which > allows reusing connection. > Why did you change the application name during the connection? -- With Regards, Amit Kapila.

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-06-27 Thread Amit Kapila
eating in patch 0003. Is it possible to avoid disconnect/connect each time the patch needs to reuse the same tablesync worker? Once, we do that and save the cost of drop_slot and associated network round trip, you may see the benefit of 0003 and 0004 patches. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-06-26 Thread Amit Kapila
On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand wrote: > > On 6/20/23 12:22 PM, Amit Kapila wrote: > > On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand > > wrote: > > >> In such a case (slot valid on the primary but invalidated on the standby) > &

Re: pg_decode_message vs skip_empty_xacts and xact_wrote_changes

2023-06-26 Thread Amit Kapila
bug. Why is pg_decode_message() > exception? > I can't see a reason why we shouldn't have a similar check for transactional messages. So, agreed this is a bug. -- With Regards, Amit Kapila.

Re: Assert while autovacuum was executing

2023-06-25 Thread Amit Kapila
On Fri, Jun 23, 2023 at 10:07 PM Andres Freund wrote: > > On 2023-06-23 14:04:15 +0530, Amit Kapila wrote: > > OTOH, if the above theory is wrong or people are not convinced, I am > > okay with removing all the changes in commits 72e78d831a and > > 3ba59ccc89. > >

Re: Assert while autovacuum was executing

2023-06-23 Thread Amit Kapila
On Thu, Jun 22, 2023 at 9:16 AM Amit Kapila wrote: > > On Wed, Jun 21, 2023 at 10:57 AM Andres Freund wrote: > > > > As far as I can tell 72e78d831a as-is is just bogus. Unfortunately that > > likely > > also means 3ba59ccc89 is not right. > > > > In

Re: Support logical replication of DDLs

2023-06-22 Thread Amit Kapila
of the statement. We may need to think about that as well. Having said that, we are still trying to write a patch to see how it looks, which may help us to jointly evaluate if we can do anything better. -- With Regards, Amit Kapila.

Re: Skip collecting decoded changes of already-aborted transactions

2023-06-22 Thread Amit Kapila
On Wed, Jun 21, 2023 at 8:12 AM Masahiko Sawada wrote: > > On Thu, Jun 15, 2023 at 7:50 PM Amit Kapila wrote: > > > > On Tue, Jun 13, 2023 at 2:06 PM Masahiko Sawada > > wrote: > > > > > > On Sun, Jun 11, 2023 at 5:31 AM Andres Freun

Re: [DOC] Update ALTER SUBSCRIPTION documentation v3

2023-06-22 Thread Amit Kapila
On Tue, Jun 20, 2023 at 9:02 AM Peter Smith wrote: > > FYI - I have created and tested back-patches for Amit's v5 patch, > going all the way to REL_10_STABLE. > Pushed. I haven't used PG10 patch as REL_10_STABLE is out of support now. -- With Regards, Amit Kapila.

Re: Assert while autovacuum was executing

2023-06-21 Thread Amit Kapila
s relation level lock is > supposed to take care of? > Yeah, I also can't see why that shouldn't be sufficient for VACUUM. Assuming your observation is correct, what do you suggest doing in this regard? -- With Regards, Amit Kapila.

Re: Assert while autovacuum was executing

2023-06-21 Thread Amit Kapila
for an index). -- With Regards, Amit Kapila.

Re: RFC: logical publication via inheritance root?

2023-06-21 Thread Amit Kapila
On Tue, Jun 20, 2023 at 10:39 PM Jacob Champion wrote: > > On Fri, Jun 16, 2023 at 9:24 PM Amit Kapila wrote: > > > The other idea that came across my mind was to provide some schema > > mapping kind of feature on subscribers where we could route the tuples > > from

Re: could not extend file "base/5/3501" with FileFallocate(): Interrupted system call

2023-06-20 Thread Amit Kapila
h + test infra, and they all passed. > > > >We still have a week till beta2 wrap, so I'd say push. If the buildfarm > >gets unhappy you can revert. > > Hah. Somehow I confused myself into thinking you're wrapping later today. > Calendar math vs Andres: 6753:3 > Can we close the open item corresponding to this after your commit 0d369ac650? -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-06-20 Thread Amit Kapila
On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand wrote: > > On 6/19/23 12:03 PM, Amit Kapila wrote: > > On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand > > wrote: > >> > >> Also I think we need to handle the case of invalidated replication > >>

Re: Assert while autovacuum was executing

2023-06-20 Thread Amit Kapila
On Mon, Jun 19, 2023 at 5:13 PM Amit Kapila wrote: > > On Sun, Jun 18, 2023 at 12:18 AM Peter Geoghegan wrote: > > > > On Sat, Jun 17, 2023 at 11:29 AM Jaime Casanova > > wrote: > > > I have been testing 16beta1, last commit > > > a14e75eb0b6a738

Re: Assert while autovacuum was executing

2023-06-19 Thread Amit Kapila
else, and > > today I found a core with the attached backtrace. > > The assertion that fails is the IsPageLockHeld assertion from commit > 72e78d831a. > I'll look into this and share my analysis. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-06-19 Thread Amit Kapila
primary/publisher should have been dropped/invalidated by that time. BTW, does the patch handles drop of logical slots on standby when the same slot is dropped on publisher/primary? -- With Regards, Amit Kapila.

Re: Support logical replication of DDLs

2023-06-19 Thread Amit Kapila
ariable name telems to telems_present to make it bit easy to follow. 7. It would be better if we can further split the patch to move Alter case into a separate patch. That will help us to focus on reviewing Create/Drop in detail. -- With Regards, Amit Kapila.

Re: Subscription statistics are not dropped at DROP SUBSCRIPTION in some cases

2023-06-18 Thread Amit Kapila
On Mon, Jun 19, 2023 at 6:50 AM Masahiko Sawada wrote: > > On Sat, Jun 17, 2023 at 6:45 PM Amit Kapila wrote: > > > > On Tue, May 16, 2023 at 8:00 PM Masahiko Sawada > > wrote: > > > > > > On Thu, May 11, 2023 at 5:12 PM Masahiko Sawada > >

Re: Subscription statistics are not dropped at DROP SUBSCRIPTION in some cases

2023-06-17 Thread Amit Kapila
backpatched to 15. > Since in pg15, since we don't create the subscription stats at CREATE > SUBSCRIPTION time but do when the first error is reported, > AFAICS, the call to pgstat_create_subscription() is present in CreateSubscription() in 15 as well, so, I don't get your point. -- With Regards, Amit Kapila.

Re: [DOC] Update ALTER SUBSCRIPTION documentation v3

2023-06-17 Thread Amit Kapila
On Fri, Jun 16, 2023 at 7:15 PM Peter Eisentraut wrote: > > On 15.06.23 04:49, Amit Kapila wrote: > > > > Now, along with this change, there is a change in errhint as well > > which I am not sure about whether to backpatch or not. I think we have > > the followi

Re: RFC: logical publication via inheritance root?

2023-06-16 Thread Amit Kapila
On Sat, Jun 17, 2023 at 1:51 AM Jacob Champion wrote: > > On Fri, Jun 16, 2023 at 6:26 AM Amit Kapila wrote: > > > > But if it's because you've implemented a partitioning scheme of your > > > own (the docs still list reasons you might want to [2], even today), >

Re: RFC: logical publication via inheritance root?

2023-06-16 Thread Amit Kapila
t by you where people want to use inheritance as a mechanism for partitioning but I feel those would still be in the minority. Personally, I am not very excited about this idea. -- With Regards, Amit Kapila.

Re: Synchronizing slots from primary to standby

2023-06-16 Thread Amit Kapila
er standby's promotion, the subscriber can start syncing from the wrong position. -- With Regards, Amit Kapila.

Re: Non-superuser subscription owners

2023-06-15 Thread Amit Kapila
On Thu, Jun 15, 2023 at 11:18 PM Alvaro Herrera wrote: > > On 2023-Jun-13, Amit Kapila wrote: > > > I'll push this tomorrow unless there are any suggestions or comments. > > Note the proposed commit message is wrong about which commit is to blame > for the origina

Re: Skip collecting decoded changes of already-aborted transactions

2023-06-15 Thread Amit Kapila
e any such reports and can we see how much is the benefit with large transactions? Because we do have the handling of concurrent aborts during sys table scans and that might help sometimes for large transactions. -- With Regards, Amit Kapila.

Re: Consistent coding for the naming of LR workers

2023-06-15 Thread Amit Kapila
makes error strings in the code look a bit shorter. I think it is better to park the patch for the next CF to avoid forgetting about it. -- With Regards, Amit Kapila.

Re: [DOC] Update ALTER SUBSCRIPTION documentation v3

2023-06-14 Thread Amit Kapila
On Wed, Jun 14, 2023 at 6:52 PM Peter Eisentraut wrote: > > On 14.06.23 05:09, Amit Kapila wrote: > > The latest version looks good to me as well. I think we should > > backpatch this change as this is a user-facing message change in docs > > and code. What do y

Re: [DOC] Update ALTER SUBSCRIPTION documentation v3

2023-06-13 Thread Amit Kapila
On Wed, Jun 14, 2023 at 9:25 AM Peter Smith wrote: > > On Wed, Jun 14, 2023 at 1:10 PM Amit Kapila wrote: > > > > On Wed, May 17, 2023 at 11:57 AM Peter Smith wrote: > > > > > > On Wed, May 17, 2023 at 2:53 PM Robert Sjöblom > > > wrote: &g

Re: [DOC] Update ALTER SUBSCRIPTION documentation v3

2023-06-13 Thread Amit Kapila
this change as this is a user-facing message change in docs and code. What do you guys think? -- With Regards, Amit Kapila.

Re: Non-superuser subscription owners

2023-06-13 Thread Amit Kapila
On Tue, Jun 13, 2023 at 2:25 PM Amit Kapila wrote: > > On Fri, May 12, 2023 at 3:28 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > I can reproduce this via gdb following similar steps in [1]. > > > > I think we need to move this call into a transaction as wel

Re: Support logical replication of DDLs

2023-06-13 Thread Amit Kapila
avoid having an additional ObjTree layer unless you or others feel we need it. I think that will reduce the overall patch size and help us to focus on one of the approaches. Surely, as suggested by you, we should also evaluate if we can generate this code for the various command types. -- With Regards, Amit Kapila.

Re: Non-superuser subscription owners

2023-06-13 Thread Amit Kapila
On Tue, Jun 13, 2023 at 2:25 PM Amit Kapila wrote: > > On Fri, May 12, 2023 at 3:28 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > I can reproduce this via gdb following similar steps in [1]. > > > > I think we need to move this call into a transaction as wel

Re: Non-superuser subscription owners

2023-06-13 Thread Amit Kapila
ssue following the steps mentioned by you and the proposed patch to fix the issue looks good to me. -- With Regards, Amit Kapila.

Re: Support logical replication of DDLs

2023-06-12 Thread Amit Kapila
On Thu, Jun 8, 2023 at 5:32 PM shveta malik wrote: > > On Thu, Jun 8, 2023 at 10:36 AM Amit Kapila wrote: > > Please find new set of patches addressing below: > a) Issue mentioned by Wang-san in [1], > b) Comments from Peter given in [2] > c) Comments from Amit given

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2023-06-12 Thread Amit Kapila
unning->nextXid)". I think the main problem is that we started decoding immediately from the point where we restored a snapshot as at that point we could have some partial running xacts. -- With Regards, Amit Kapila.

Re: Support logical replication of DDLs

2023-06-08 Thread Amit Kapila
if we want to optimize the default mode such that we only WAL log the DDL with user-specified options then appending options for default GUCs would make the string to be WAL logged unnecessarily long. I am thinking that in default mode we can allow subscribers to perform operations with their default respective GUCs. -- With Regards, Amit Kapila.

Re: running logical replication as the subscription owner

2023-06-08 Thread Amit Kapila
On Thu, Jun 8, 2023 at 6:32 AM Masahiko Sawada wrote: > > On Mon, Jun 5, 2023 at 3:15 AM Amit Kapila wrote: > > > > On Fri, May 26, 2023 at 6:18 PM Masahiko Sawada > > wrote: > > > > > > On Thu, May 25, 2023 at 5:41 PM Amit Kapila > > >

Re: Support logical replication of DDLs

2023-06-07 Thread Amit Kapila
On Tue, Jun 6, 2023 at 4:26 PM Amit Kapila wrote: > > > Few assorted comments: > === > Few comments on 0008* patch: == 1. After 0008*, deparse_CreateStmt(), forms dpcontext before forming identity whereas it is required only after it. It

Re: Reload configuration more frequently in apply worker.

2023-06-07 Thread Amit Kapila
On Tue, Jun 6, 2023 at 7:45 AM Zhijie Hou (Fujitsu) wrote: > > On Monday, June 5, 2023 8:03 AM Amit Kapila wrote: > > > > > > I am able to reproduce the problem with the script and steps mentioned by > > you. The patch provided by you fixes the problem and l

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2023-06-06 Thread Amit Kapila
alysis/thoughts. BTW, do you mean that it skips the assignment (a) because the assignment record is before we reach a consistent point, or (b) because we start reading WAL after the assignment, or (c) something else? If you intend to say (a) then can you please point me to the code you are referring to for the same? -- With Regards, Amit Kapila.

Re: Support logical replication of DDLs

2023-06-06 Thread Amit Kapila
/test/modules/test_ddl_deparse_regress/t/001_compare_dumped_results.pl.orig Will this file require for the 0008 patch? Or is this just a leftover? -- With Regards, Amit Kapila.

Re: Implement generalized sub routine find_in_log for tap test

2023-06-05 Thread Amit Kapila
refactoring for v16. However, sometimes, we do decide to backpatch refactoring in tests to avoid backpatch effort. I am not completely sure if that is the case here. -- With Regards, Amit Kapila.

Re: Reload configuration more frequently in apply worker.

2023-06-04 Thread Amit Kapila
On Wed, May 17, 2023 at 8:34 AM Amit Kapila wrote: > > On Wed, May 17, 2023 at 7:18 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > The attached tap test shows how the failure happened. > > > > I haven't yet tried to reproduce it but will try later sometime

Re: running logical replication as the subscription owner

2023-06-04 Thread Amit Kapila
On Fri, May 26, 2023 at 6:18 PM Masahiko Sawada wrote: > > On Thu, May 25, 2023 at 5:41 PM Amit Kapila wrote: > > I've attached the updated patch. Please review it. > Few comments: 1. + /* get the owner for ACL and RLS checks */ + run_as_owner = MySubscription->runaso

Re: Wrong syntax in feature description

2023-06-04 Thread Amit Kapila
ot;ALL", see [2]) > Right, this should be changed. -- With Regards, Amit Kapila.

Re: running logical replication as the subscription owner

2023-05-25 Thread Amit Kapila
On Thu, May 25, 2023 at 12:33 PM Masahiko Sawada wrote: > > On Tue, May 23, 2023 at 8:21 PM Amit Kapila wrote: > > > > On Mon, May 22, 2023 at 6:06 PM Masahiko Sawada > > wrote: > > > > > > Thank you for updating the patch! Here are review comments: &g

Re: running logical replication as the subscription owner

2023-05-23 Thread Amit Kapila
the permission to the target table. The > test passes only if run_as_owner = true works. > Why in the test do you need to give additional permissions to regress_admin2 when the actual operation has to be performed by the table owner? +# Because the initial data sync is working as the t

Re: Initial Schema Sync for Logical Replication

2023-05-22 Thread Amit Kapila
feature complex > and I'm not sure users can use it properly. > > Third idea is that since the use case of synchronizing the whole > database can be achievable even by pg_dump(all), we support > synchronizing only tables (+ indexes) in the initial sync feature, > which can not b

Re: Possible regression setting GUCs on \connect

2023-05-17 Thread Amit Kapila
further discussion, it won't be easy to conclude that. Tom/Nathan, do you have any further suggestions here? [1] - https://www.postgresql.org/message-id/capphfdsld6e--epngqxenqlp6dlwunzrpmcnyb3wj87wr7u...@mail.gmail.com -- With Regards, Amit Kapila.

Re: Reload configuration more frequently in apply worker.

2023-05-16 Thread Amit Kapila
(it's very rare and only happened once in 4 > months) which I think is due to the low frequent reload in apply worker. > > The attached tap test shows how the failure happened. > I haven't yet tried to reproduce it but will try later sometime. Thanks for your analysis. [1] - https://www.postgresql.org/message-id/2138662.1623460441%40sss.pgh.pa.us -- With Regards, Amit Kapila.

Re: running logical replication as the subscription owner

2023-05-16 Thread Amit Kapila
eration on a particular table, it will switch roles to the table owner and perform the operation with the table owner's privileges." to be bit more specific about initial sync process as well? [1] - https://www.postgresql.org/docs/devel/logical-replication-security.html -- With Regards, Amit Kapila.

Re: running logical replication as the subscription owner

2023-05-16 Thread Amit Kapila
On Fri, May 12, 2023 at 5:25 PM Ajin Cherian wrote: > > On Fri, May 12, 2023 at 1:49 PM Amit Kapila wrote: > > > > I tried the following test: > > > Repeat On the publisher and subscriber: > /* Create role regress_alice with NOSUPERUSER on

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-05-11 Thread Amit Kapila
On Fri, May 12, 2023 at 10:33 AM Amit Kapila wrote: > > On Fri, May 12, 2023 at 7:38 AM Masahiko Sawada wrote: > > > > On Thu, May 11, 2023 at 2:04 PM Amit Kapila wrote: > > > > > > On Fri, Apr 28, 2023 at 2:35 PM Hayato Kuroda (Fujitsu) > &

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-05-11 Thread Amit Kapila
On Fri, May 12, 2023 at 7:38 AM Masahiko Sawada wrote: > > On Thu, May 11, 2023 at 2:04 PM Amit Kapila wrote: > > > > On Fri, Apr 28, 2023 at 2:35 PM Hayato Kuroda (Fujitsu) > > wrote: > > > > > > Dear hackers, > > > > > &

Re: running logical replication as the subscription owner

2023-05-11 Thread Amit Kapila
On Fri, May 12, 2023 at 9:10 AM Masahiko Sawada wrote: > > On Fri, May 12, 2023 at 1:12 AM Robert Haas wrote: > > > > On Thu, May 11, 2023 at 7:38 AM Amit Kapila wrote: > > > Do we want the initial sync to also respect 'run_as_owner' option? I > > > might

Re: running logical replication as the subscription owner

2023-05-11 Thread Amit Kapila
be missing something but I don't see anything in the docs about initial sync interaction with this option. In the commit a2ab9c06ea, we did the permission checking during the initial sync so I thought we should do it here as well. -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-05-10 Thread Amit Kapila
2D%3DeuO-Qs-GAg%40mail.gmail.com -- With Regards, Amit Kapila.

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-05-10 Thread Amit Kapila
da-San came up with a new patch with a different design based on the discussion here. I haven't looked at it yet though. [1] - https://about.gitlab.com/blog/2019/02/13/delayed-replication-for-disaster-recovery-with-postgresql/ [2] - https://dev.mysql.com/doc/refman/8.0/en/replication-delayed.html [3] - https://www.postgresql.org/message-id/73b06a32-56ab-4056-86ff-e307f3c316f1%40www.fastmail.com -- With Regards, Amit Kapila.

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-10 Thread Amit Kapila
On Tue, May 9, 2023 at 12:44 PM Drouvot, Bertrand wrote: > > On 5/9/23 8:02 AM, Amit Kapila wrote: > > On Mon, May 8, 2023 at 1:45 PM Drouvot, Bertrand > > wrote: > > > > > Why not initialize the cascading standby node just before the standby > >

Re: walsender performance regression due to logical decoding on standby changes

2023-05-10 Thread Amit Kapila
SndWakeup(switchedTLI, true); > + ConditionVariableBroadcast(>cv); > > After the change, we wakeup physical walsender regardless of switchedTLI flag. > Is this intentional ? if so, I think It would be better to update the > comments above this. > This raises the question of whether we need this condition variable logic for physical walsenders? -- With Regards, Amit Kapila.

Re: Tables getting stuck at 's' state during logical replication

2023-05-09 Thread Amit Kapila
appen because apply worker should eventually update these relations state to 'r'. If this is reproducible, we can try two things to further investigate the issue: (a) Disable and Enable the subscription once and see if that helps; (b) try increasing the LOG level to DEBUG2 and see if we get any useful LOG message. -- With Regards, Amit Kapila.

Re: Perform streaming logical transactions by background workers and parallel apply

2023-05-09 Thread Amit Kapila
On Tue, May 9, 2023 at 7:50 AM Masahiko Sawada wrote: > > On Mon, May 8, 2023 at 8:09 PM Amit Kapila wrote: > > > > > > I think it is only possible for the leader apply can worker to try to > > receive the error message from an error queue after your 0002 patch. >

Re: [DOC] Update ALTER SUBSCRIPTION documentation

2023-05-09 Thread Amit Kapila
t_name=NONE. It would be better to discuss this in a separate thread to seek the opinion of others. -- With Regards, Amit Kapila.

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-09 Thread Amit Kapila
On Mon, May 8, 2023 at 1:45 PM Drouvot, Bertrand wrote: > > On 5/8/23 4:42 AM, Amit Kapila wrote: > > On Sat, May 6, 2023 at 9:33 PM Drouvot, Bertrand > > wrote: > >> > >> On 5/6/23 3:28 PM, Amit Kapila wrote: > >>> On Sat, May 6, 2023 at 1:52 PM

<    5   6   7   8   9   10   11   12   13   14   >