RE: Conflict detection for update_deleted in logical replication

2025-09-20 Thread Zhijie Hou (Fujitsu)
On Friday, September 12, 2025 4:43 PM Amit Kapila wrote: > > On Fri, Sep 12, 2025 at 8:55 AM Zhijie Hou (Fujitsu) > wrote: > > > > I agree. Here is a V73 patch that will restart the worker if the > > retention resumes. I also addressed other comments posted by Ami

RE: Conflict detection for update_deleted in logical replication

2025-09-20 Thread Zhijie Hou (Fujitsu)
On Wednesday, September 3, 2025 12:19 PM shveta malik wrote: > On Tue, Sep 2, 2025 at 3:30 PM shveta malik > wrote: > > > > > > > > > > > > > Here is V70 patch set. > > > > > > > > > Please find a few comments on v70-003: > > 1) > Doc of dead_tuple_retention_active says: > True if retain_dead_

RE: Conflict detection for update_deleted in logical replication

2025-09-20 Thread Zhijie Hou (Fujitsu)
On Monday, September 8, 2025 7:21 PM Zhijie Hou (Fujitsu) wrote: > > On Monday, September 8, 2025 3:13 PM Amit Kapila > wrote: > > > > On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) > > > > wrote: > > > > > > Here are v2 patches which ad

RE: Parallel Apply

2025-09-16 Thread Zhijie Hou (Fujitsu)
On Wednesday, September 17, 2025 2:40 AM Konstantin Knizhnik wrote: > On 11/08/2025 7:45 AM, Amit Kapila wrote: > > 4. Triggers and Constraints For the initial version, exclude tables with > > user-defined triggers or constraints from parallel apply due to complexity > > in > > dependency detec

RE: Reword messages using "as" instead of "because"

2025-09-16 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 16, 2025 11:56 AM Amit Kapila wrote: > > On Tue, Sep 16, 2025 at 8:17 AM Kyotaro Horiguchi > wrote: > > > > I noticed that the recent commit 0d48d393d46 introduced the following > > three messages: > > > > 4793> errdetail("Retention is stopped as the apply process is not >

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 16, 2025 11:54 AM Zhijie Hou (Fujitsu) wrote: > > On Monday, September 15, 2025 8:11 PM Amit Kapila > wrote: > > > > On Mon, Sep 15, 2025 at 1:07 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Thanks, the changes loo

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Monday, September 15, 2025 8:11 PM Amit Kapila wrote: > > On Mon, Sep 15, 2025 at 1:07 PM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks, the changes look good to me. I have merged them in V75 patch. > > > > Pushed. I see a BF which is not related with this c

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Monday, September 15, 2025 12:52 PM Amit Kapila wrote: > > On Fri, Sep 12, 2025 at 3:39 PM Zhijie Hou (Fujitsu) > wrote: > > > > Here is the V74 patch which addressed all comments. > > > > + ereport(LOG, > + errmsg("logical replication worker for su

RE: Conflict detection for update_deleted in logical replication

2025-09-15 Thread Zhijie Hou (Fujitsu)
On Monday, September 15, 2025 12:55 PM shveta malik wrote: > > One concern: > > if (should_stop_conflict_info_retention(rdt_data)) > + { > + /* > + * Stop retention if not yet. Otherwise, reset to the initial phase to > + * retry all phases. This is required to recalculate the current wait >

RE: Conflict detection for update_deleted in logical replication

2025-09-12 Thread Zhijie Hou (Fujitsu)
On Friday, September 12, 2025 4:48 PM shveta malik wrote: > On Fri, Sep 12, 2025 at 8:55 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > I agree. Here is a V73 patch that will restart the worker if the > > retention resumes. I also addressed other comments posted b

RE: Conflict detection for update_deleted in logical replication

2025-09-12 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 2, 2025 6:00 PM shveta malik wrote: > > On Mon, Sep 1, 2025 at 5:45 PM shveta malik > wrote: > > > > > > > > Here is V70 patch set. > > > > > > > The patch v70-0001 looks good to me. Verified, all the old issues are > > resolved. > > > > Will resume review of v70-0002 now

RE: Add support for specifying tables in pg_createsubscriber.

2025-09-11 Thread Zhijie Hou (Fujitsu)
On Wednesday, September 3, 2025 9:58 AM Peter Smith wrote: > > Hi Shubham, > > My first impression of patch v4-0001 was that the new design is adding > unnecessary complexity by introducing a new --existing-publication > option, instead of just allowing reinterpretation of the --publication > b

RE: Conflict detection for update_deleted in logical replication

2025-09-11 Thread Zhijie Hou (Fujitsu)
On Friday, September 12, 2025 2:39 AM Masahiko Sawada wrote: > > On Thu, Sep 11, 2025 at 3:54 AM Dilip Kumar wrote: > > > > On Wed, Sep 10, 2025 at 2:09 PM Amit Kapila > wrote: > > > > > > On Wed, Sep 10, 2025 at 9:08 AM Zhijie Hou (Fujitsu) > > &g

RE: Conflict detection for update_deleted in logical replication

2025-09-09 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 9, 2025 7:01 PM Amit Kapila wrote: > On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) > wrote: > > > > Here is V71 patch set which addressed above comments and [1]. > > > > IIUC, this patch after stopping the retention, it immediately sta

RE: Conflict detection for update_deleted in logical replication

2025-09-09 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 9, 2025 5:17 PM shveta malik wrote: > > On Tue, Sep 9, 2025 at 11:47 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > Here is V71 patch set which addressed above comments and [1]. > > > > Thank You for the patches. Please f

RE: issue with synchronized_standby_slots

2025-09-08 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 9, 2025 1:30 PM Amit Kapila wrote: > > On Mon, Sep 8, 2025 at 2:56 PM Alexander Kukushkin > wrote: > > > > Recently we also hit this problem. > > > > I think in a current state check_synchronized_standby_slots() and > validate_sync_standby_slots() functions are not very us

RE: Conflict detection for update_deleted in logical replication

2025-09-08 Thread Zhijie Hou (Fujitsu)
On Monday, September 8, 2025 3:13 PM Amit Kapila wrote: > > On Fri, Sep 5, 2025 at 5:03 PM Zhijie Hou (Fujitsu) > wrote: > > > > Here are v2 patches which addressed above comments. > > > > I have pushed the first patch. I find that the test can't rel

RE: Avoid retaining conflict-related data when no tables are subscribed

2025-09-05 Thread Zhijie Hou (Fujitsu)
On Thursday, September 4, 2025 7:38 PM Amit Kapila wrote: > > On Tue, Sep 2, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Friday, August 29, 2025 12:05 PM Amit Kapila > wrote: > > > > > > bool > > > -AllTablesyncsReady(void

RE: Conflict detection for update_deleted in logical replication

2025-09-05 Thread Zhijie Hou (Fujitsu)
On Friday, September 5, 2025 2:01 PM shveta malik wrote: > > On Thu, Sep 4, 2025 at 3:30 PM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > As reported by Robert[1], it is worth adding a test for the race condition > > in > > the RecordTransactio

RE: issue with synchronized_standby_slots

2025-09-05 Thread Zhijie Hou (Fujitsu)
On Thursday, September 4, 2025 9:27 PM Fabrice Chapuis wrote: > With PG 17.5 and using logical replication failover slots. When trying to > change the value of synchronized_standby_slots, node2 was not running then the > error invalid value for parameter "synchronized_standby_slots": "node1,nod

RE: Conflict detection for update_deleted in logical replication

2025-09-04 Thread Zhijie Hou (Fujitsu)
Hi, As reported by Robert[1], it is worth adding a test for the race condition in the RecordTransactionCommitPrepared() function to reduce the risk of future code changes: /* * Note it is important to set committs value after marking ourselves as * in the commit critical

RE: pgsql: Preserve conflict-relevant data during logical replication.

2025-09-03 Thread Zhijie Hou (Fujitsu)
On Tuesday, September 2, 2025 11:03 PM Robert Haas wrote: > > On Wed, Jul 23, 2025 at 11:44 PM Amit Kapila > wrote: > > The fix looks good to me. I'll push your patch in sometime. > > The tests in this patch are insufficient to prove that this logic works > properly. I > tried with this patch

RE: Avoid retaining conflict-related data when no tables are subscribed

2025-09-02 Thread Zhijie Hou (Fujitsu)
On Friday, August 29, 2025 12:05 PM Amit Kapila wrote: > > On Thu, Aug 28, 2025 at 7:54 AM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > My colleague Nisha reported an issue to me off-list: dead tuples can't > > be removed when retain_dead_

RE: Conflict detection for update_deleted in logical replication

2025-08-31 Thread Zhijie Hou (Fujitsu)
On Saturday, August 30, 2025 12:48 PM Nisha Moond wrote: > > On Fri, Aug 29, 2025 at 11:49 AM Zhijie Hou (Fujitsu) > wrote: > > > > Here is the new version patch set which also addressed Shveta's > comments[1]. > > > > Thanks for the patches here, I te

RE: Conflict detection for update_deleted in logical replication

2025-08-28 Thread Zhijie Hou (Fujitsu)
On Thursday, August 28, 2025 6:07 PM shveta malik wrote: > On Thu, Aug 28, 2025 at 8:02 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > I noticed that Cfbot failed to compile the document due to a typo > > after renaming the subscription option. Here are the updat

Avoid retaining conflict-related data when no tables are subscribed

2025-08-27 Thread Zhijie Hou (Fujitsu)
Hi, My colleague Nisha reported an issue to me off-list: dead tuples can't be removed when retain_dead_tuples is enabled for a subscription with no tables. This appears to stem from the inability to advance the non-removable transaction ID when AllTablesyncsReady() returns false. Since this funct

RE: Conflict detection for update_deleted in logical replication

2025-08-26 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 26, 2025 2:45 PM Amit Kapila wrote: > On Mon, Aug 25, 2025 at 5:05 PM Amit Kapila > wrote: > > > > A few comments on 0001: > > > > Some more comments: Thanks for the comments! > 1. > + /* > + * Return false if the leader apply worker has stopped retaining > + * information f

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-25 Thread Zhijie Hou (Fujitsu)
On Monday, August 25, 2025 8:08 AM Peter Smith wrote: > > IIUC, the only purpose of the proposed '--table' spec is to allow some users > the > ability to tweak the default "CREATE PUBLICATION p FOR ALL TABLES;" that > the 'pg_createsubscriber' tool would otherwise construct. > > But --table is

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-25 Thread Zhijie Hou (Fujitsu)
On Friday, August 22, 2025 11:26 PM Euler Taveira wrote: > > On Fri, Aug 22, 2025, at 6:57 AM, Zhijie Hou (Fujitsu) wrote: > > The documentation appears incorrect and needs revision. The latest > > version no longer depends on the option order; instead, it requires > > u

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-22 Thread Zhijie Hou (Fujitsu)
On Friday, August 22, 2025 11:19 AM Euler Taveira wrote: > > On Thu, Aug 21, 2025, at 6:08 AM, Shubham Khanna wrote: > > Attachments: > > * v3-0001-Support-tables-via-pg_createsubscriber.patch > > * v3-0002-Support-WHERE-clause-and-COLUMN-list-in-table-arg.patch > > + --table= class="paramet

RE: Conflict detection for update_deleted in logical replication

2025-08-21 Thread Zhijie Hou (Fujitsu)
On Thursday, August 21, 2025 2:01 PM shveta malik wrote: > > On Wed, Aug 20, 2025 at 12:12 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > I agree. Here is V63 version which implements this approach. > > > > Thank You for the patches. > &

RE: Fix replica identity checks for MERGE command on published table.

2025-08-20 Thread Zhijie Hou (Fujitsu)
On Friday, July 11, 2025 3:09 PM Dean Rasheed wrote: > > On Tue, 8 Jul 2025 at 09:51, Dean Rasheed > wrote: > > > > Answering my own question, INSERT ... ON CONFLICT DO UPDATE does > have > > the same problem as MERGE. To reproduce the error, all you need to do > > is create the unique index it

RE: Conflict detection for update_deleted in logical replication

2025-08-18 Thread Zhijie Hou (Fujitsu)
On Monday, August 18, 2025 2:32 PM Dilip Kumar wrote: > > On Mon, Aug 18, 2025 at 10:36 AM Amit Kapila > wrote: > > > > > --- > > > Even if an apply worker disables retaining dead tuples due to > > > max_conflict_retention_duration, it enables again after the server > > > restarts. > > > > > > >

RE: Conflict detection for update_deleted in logical replication

2025-08-17 Thread Zhijie Hou (Fujitsu)
On Saturday, August 16, 2025 7:44 AM Masahiko Sawada wrote: > Here are review comments on v62 patch: Thanks for the comments! > > > --- > +else if (IsSet(supported_opts, > SUBOPT_MAX_CONFLICT_RETENTION_DURATION) && > + strcmp(defel->defname, > "

RE: memory leak in logical WAL sender with pgoutput's cachectx

2025-08-14 Thread Zhijie Hou (Fujitsu)
On Friday, August 15, 2025 10:59 AM Xuneng Zhou wrote: > Thanks for your clarification! > > On Fri, Aug 15, 2025 at 10:10 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Thursday, August 14, 2025 8:49 PM Hayato Kuroda (Fujitsu) > wrote: > > > > > > Dear

RE: memory leak in logical WAL sender with pgoutput's cachectx

2025-08-14 Thread Zhijie Hou (Fujitsu)
On Thursday, August 14, 2025 8:49 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Xuneng, > > > Is it safe to free the substructure from within > > rel_sync_cache_relation_cb()? > > You referred the comment in rel_sync_cache_relation_cb() right? I understood > like that we must not access to any *

RE: Parallel Apply

2025-08-13 Thread Zhijie Hou (Fujitsu)
On Monday, August 11, 2025 12:46 PM Amit Kapila wrote: > Background and Motivation > - > In high-throughput systems, where hundreds of sessions generate data > on the publisher, the subscriber's apply process often becomes a > bottleneck due to the single apply

RE: Conflict detection for update_deleted in logical replication

2025-08-12 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 12, 2025 5:01 PM Dilip Kumar wrote: Hi, > > On Tue, Aug 12, 2025 at 2:21 PM Amit Kapila > wrote: > > > > On Tue, Aug 12, 2025 at 2:06 PM shveta malik > wrote: > > > > > > 2) > > > postgres=# create subscription sub2 connection 'dbname=postgres > > > host=localhost user=shv

RE: Conflict detection for update_deleted in logical replication

2025-08-08 Thread Zhijie Hou (Fujitsu)
On Friday, August 8, 2025 2:34 PM shveta malik wrote: > > On Thu, Aug 7, 2025 at 10:10 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Tuesday, August 5, 2025 10:09 AM Zhijie Hou (Fujitsu) > wrote: > > > Here is V57 patch set which addressed most of comments. &g

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-08 Thread Zhijie Hou (Fujitsu)
On Wednesday, August 6, 2025 7:23 PM vignesh C wrote: > On Fri, 1 Aug 2025 at 13:33, Zhijie Hou (Fujitsu) > wrote: > > > > On Monday, July 28, 2025 1:07 PM Hayato Kuroda (Fujitsu) > wrote: > > > > > > Dear Shubham, > > > > > > > The

RE: Conflict detection for update_deleted in logical replication

2025-08-06 Thread Zhijie Hou (Fujitsu)
On Tuesday, August 5, 2025 10:09 AM Zhijie Hou (Fujitsu) wrote: > Here is V57 patch set which addressed most of comments. > > In this version, I also fixed a bug that the apply worker continued to find > dead > tuples even if it has already stop retaining dead tuples. Here is

RE: Conflict detection for update_deleted in logical replication

2025-08-04 Thread Zhijie Hou (Fujitsu)
On Monday, August 4, 2025 2:16 PM shveta malik wrote: > > On Fri, Aug 1, 2025 at 9:16 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > Thanks for confirming. Here is V56 patch set which addressed all the > > comments including the comments from Amit[1] and Shveta[2

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Saturday, August 2, 2025 12:59 AM Andrew Dunstan wrote: > On 2025-08-01 Fr 11:03 AM, Zhijie Hou (Fujitsu) wrote: > > On Friday, August 1, 2025 8:56 PM Andrew Dunstan mailto:and...@dunslane.net > wrote: > > > > > We have another example to consider: pg_amc

RE: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Friday, August 1, 2025 7:42 PM Dilip Kumar wrote: > > On Fri, Aug 1, 2025 at 5:02 PM Zhijie Hou (Fujitsu) > wrote: > > > > > 2. > > > > > > + If set to true, the detection of > > > + is enabled, and > > >

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Friday, August 1, 2025 8:56 PM Andrew Dunstan wrote: > On 2025-08-01 Fr 4:03 AM, Zhijie Hou (Fujitsu) wrote: > > On Monday, July 28, 2025 1:07 PM Hayato Kuroda (Fujitsu) > > mailto:kuroda.hay...@fujitsu.com wrote: > > > Dear Shubham, > > > > > > Th

RE: Conflict detection for update_deleted in logical replication

2025-08-01 Thread Zhijie Hou (Fujitsu)
t detection for update_deleted in logical replication > > On Thu, Jul 31, 2025 at 3:49 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Thursday, July 31, 2025 5:29 PM Amit Kapila > wrote: > > > > > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) >

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 1:07 PM Hayato Kuroda (Fujitsu) wrote: > > Dear Shubham, > > > The attached patch introduces a new '--table' option that can be > > specified after each '--database' argument. > > Do we have another example which we consider the ordering of options? I'm > unsure > for

RE: Add support for specifying tables in pg_createsubscriber.

2025-08-01 Thread Zhijie Hou (Fujitsu)
On Monday, July 21, 2025 1:31 PM Shubham Khanna wrote: > > Hi hackers, > > Currently, pg_createsubscriber supports converting streaming > replication to logical replication for selected databases or all > databases. However, there is no provision to replicate only a few > selected tables. For s

RE: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Zhijie Hou (Fujitsu)
On Thursday, July 31, 2025 5:26 PM shveta malik wrote: > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > wrote: > > > > > > This is the V54 patch set, with only patch 0001 updated to address the > > latest comments. > > > > Thanks for

RE: Conflict detection for update_deleted in logical replication

2025-07-31 Thread Zhijie Hou (Fujitsu)
On Thursday, July 31, 2025 5:29 PM Amit Kapila wrote: > > On Tue, Jul 29, 2025 at 10:51 AM Zhijie Hou (Fujitsu) > wrote: > > > > This is the V54 patch set, with only patch 0001 updated to address the > > latest comments. > > > > Few minor comments: Than

RE: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 5:54 PM Amit Kapila wrote: > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > wrote: > > > > Right, I think it makes sense to do with the index scan when the > > index's xmin is less than the conflict detection xmin, as that ca

RE: Conflict detection for update_deleted in logical replication

2025-07-28 Thread Zhijie Hou (Fujitsu)
On Monday, July 28, 2025 7:43 PM shveta malik wrote: > On Mon, Jul 28, 2025 at 4:38 PM shveta malik > wrote: > > > > On Fri, Jul 25, 2025 at 4:38 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > > > > The V53-0001 also includes Shveta'

RE: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Zhijie Hou (Fujitsu)
On Friday, July 25, 2025 2:33 PM Amit Kapila wrote: > > On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) > wrote: > > > > Thanks for pushing. I have rebased the remaining patches. > > > > + * This function performs a full table scan instead of using inde

RE: Conflict detection for update_deleted in logical replication

2025-07-25 Thread Zhijie Hou (Fujitsu)
On Thursday, July 24, 2025 11:42 AM shveta malik wrote: > > On Wed, Jul 23, 2025 at 12:53 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wednesday, July 23, 2025 12:08 PM Amit Kapila > wrote: > > > > > > On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada &

RE: Conflict detection for update_deleted in logical replication

2025-07-23 Thread Zhijie Hou (Fujitsu)
On Wednesday, July 23, 2025 12:08 PM Amit Kapila wrote: > > On Wed, Jul 23, 2025 at 3:51 AM Masahiko Sawada > wrote: > > > > I've reviewed the 0001 patch and it looks good to me. > > > > Thanks, I have pushed the 0001 patch. Thanks for pushing. I have rebased the remaining patches. I have re

RE: Logical replication prefetch

2025-07-14 Thread Zhijie Hou (Fujitsu)
On Monday, July 14, 2025 2:36 PM Konstantin Knizhnik wrote: > On 14/07/2025 4:20 am, Zhijie Hou (Fujitsu) wrote: > > Additionally, I was also exploring ways to improve performance and have > > tried an > > alternative version of prefetch for experimentation. The alter

RE: Logical replication prefetch

2025-07-13 Thread Zhijie Hou (Fujitsu)
On Tuesday, July 8, 2025 2:36 PM Konstantin Knizhnik wrote: > > There is well known Postgres problem that logical replication subscriber can > not caught-up with publisher just because LR changes are applied by single > worker and at publisher changes are made by multiple concurrent backends. > T

RE: Conflict detection for update_deleted in logical replication

2025-07-07 Thread Zhijie Hou (Fujitsu)
On Mon, Jul 7, 2025 at 11:03 AM Zhijie Hou (Fujitsu) wrote: > > On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > > == > > > == > > > The workload is mostly same as [4]. > >

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Zhijie Hou (Fujitsu)
On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > On Sun, Jul 6, 2025 at 8:03 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Dear hackers, > > > > As a confirmation purpose, I did performance testing with four > > workloads we did before. > > Thank you for doing the performance tests! >

RE: Conflict detection for update_deleted in logical replication

2025-07-06 Thread Zhijie Hou (Fujitsu)
On Sun, Jul 6, 2025 at 10:51 PM Masahiko Sawada wrote: > > On Fri, Jul 4, 2025 at 8:18 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Jul 2, 2025 at 3:28 PM Hou, Zhijie wrote: > > > Kindly use the latest patch set for performance testing. > > > > D

RE: Report replica identity in pg_publication_tables

2025-07-02 Thread Zhijie Hou (Fujitsu)
On Mon, Jun 30, 2025 at 8:09 PM Ashutosh Bapat wrote: > > On Mon, Jun 30, 2025 at 5:17 PM Amit Kapila > wrote: > > > > On Mon, Jun 30, 2025 at 3:44 PM Ashutosh Bapat > > wrote: > > > > > Given a set of publications that a WAL sender is using, > > > pg_publication_tables can be used to get the

RE: Invalid pointer access in logical decoding after error

2025-07-02 Thread Zhijie Hou (Fujitsu)
On Wed, Jul 2, 2025 at 2:42 PM vignesh C wrote: > > Hi, > > I encountered an invalid pointer access issue. Below are the steps to > reproduce the issue: ... > The error occurs because entry->columns is allocated in the entry > private context (entry->entry_cxt) by pub_collist_to_bitmapset(). T

RE: Conflict detection for update_deleted in logical replication

2025-07-01 Thread Zhijie Hou (Fujitsu)
On Tue, Jul 1, 2025 at 6:10 PM Zhijie Hou (Fujitsu) wrote: > Here is V45 patch set. With the main patch set now stable, I am summarizing the performance tests conducted before for reference. In earlier tests [1], we confirmed that in a pub-sub cluster with high workload on the publisher (

RE: Conflict detection for update_deleted in logical replication

2025-06-25 Thread Zhijie Hou (Fujitsu)
On Wed, Jun 25, 2025 at 2:57 PM shveta malik wrote: > > > > > Here is the V41 patch set which includes the following changes: > > > > Thanks for the patches. Few trivial things: > > 1) > In ReplicationSlotAcquire(), does it make more sense to move the error after > checking the slot's existence

RE: Slot's restart_lsn may point to removed WAL segment after hard restart unexpectedly

2025-06-24 Thread Zhijie Hou (Fujitsu)
Hi, After commit ca307d5, I noticed another crash when testing some other logical replication features. The server with max_replication_slots set to 0 would crash when executing CHECKPOINT. TRAP: failed Assert("ReplicationSlotCtl != NULL"), File: "slot.c", Line: 1162, PID: 577315 postgres: che

RE: Replication slot is not able to sync up

2025-06-15 Thread Zhijie Hou (Fujitsu)
On Sat, Jun 14, 2025 at 11:37 PM Dilip Kumar wrote: > > On Fri, May 30, 2025 at 3:38 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, May 28, 2025 at 2:09 AM Masahiko Sawada wrote: > > > > > > On Fri, May 23, 2025 at 10:07 PM Amit Kapila > > > &g

RE: Logical Replication slot disappeared after promote Standby

2025-06-12 Thread Zhijie Hou (Fujitsu)
On Thu, Jun 12, 2025 at 4:08 PM Perumal Raj wrote: > Hi Community, > > I have installed postgres version 17.5 with following setup, > > Primary >-- Secondary A >-- Secondary B > -- Secondary C > > Config: > wal_level = 'logical' > max_wal_senders = '10' > max

RE: failover logical replication slots

2025-06-10 Thread Zhijie Hou (Fujitsu)
On Tue, Jun 10, 2025 at 11:46 PM Fabrice Chapuis wrote: > I'm working with logical replication in a PostgreSQL 17 setup, and I'm > exploring the new option to make replication slots failover safe in a highly > available environment using physical standby nodes managed by Patroni. > > After a swit

RE: Replication slot is not able to sync up

2025-06-10 Thread Zhijie Hou (Fujitsu)
On Thu, May 29, 2025 at 11:09 AM shveta malik wrote: > > On Wed, May 28, 2025 at 11:56 AM Masahiko Sawada > wrote: > > > > > > I didn't know it was intended for testing and debugging purposes so > > clearilying it in the documentation would be a good idea. > > I have added the suggested docs i

RE: Conflict detection for update_deleted in logical replication

2025-06-04 Thread Zhijie Hou (Fujitsu)
On Wed, Jun 4, 2025 at 6:43 PM Zhijie Hou (Fujitsu) wrote: > > On Mon, Jun 2, 2025 at 2:39 PM Amit Kapila wrote: > > > > On Mon, May 26, 2025 at 12:46 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > Attaching the V32 patch set which addressed comments

RE: Replication slot is not able to sync up

2025-05-30 Thread Zhijie Hou (Fujitsu)
On Wed, May 28, 2025 at 2:09 AM Masahiko Sawada wrote: > > On Fri, May 23, 2025 at 10:07 PM Amit Kapila > wrote: > > > > In the case presented here, the logical slot is expected to keep > > forwarding, and in the consecutive sync cycle, the sync should be > > successful. Users using logical decod

RE: Replication slot is not able to sync up

2025-05-27 Thread Zhijie Hou (Fujitsu)
On Wed, May 28, 2025 at 2:09 AM Masahiko Sawada wrote: > > On Fri, May 23, 2025 at 10:07 PM Amit Kapila > wrote: > > > > In the case presented here, the logical slot is expected to keep > > forwarding, and in the consecutive sync cycle, the sync should be > > successful. Users using logical decod

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Fri, May 23, 2025 at 11:51 PM Xuneng Zhou wrote: > Thanks for the effort on the patches. I did a quick look on them before > diving into the logic and discussion. Below are a few minor typos found in > version 31. Thanks for the comments! I have fixed these typos in latest version. Best Regar

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Fri, May 23, 2025 at 2:45 PM shveta malik wrote: > > Thanks you for v31 patch-set. Please find few comments on patch001: Thanks for the comments. > > 1) > > wait_for_local_flush: > > + if (data->last_recv_time && > + TimestampDifferenceExceeds(data->flushpos_update_time, > +data->last

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Sat, May 24, 2025 at 6:28 PM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 11:00 AM Amit Kapila > wrote: > > > > On Sat, May 24, 2025 at 10:29 AM Dilip Kumar > wrote: > > > > > > On Sat, May 24, 2025 at 10:04 AM Dilip Kumar > wrote: > > > > > > > > On Fri, May 23, 2025 at 9:21 PM Xuneng Zh

RE: Conflict detection for update_deleted in logical replication

2025-05-26 Thread Zhijie Hou (Fujitsu)
On Sun, May 25, 2025 at 4:36 PM Dilip Kumar wrote: > > On Sat, May 24, 2025 at 4:46 PM Amit Kapila > wrote: > > > > This sounds reasonable to me. Let us see what others think. > > > > I was looking into the for getting the transaction status from > publisher, what I would assume this patch shou

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Fri, May 16, 2025 at 7:31 PM Amit Kapila wrote: > > A few more comments: > = > 3. > maybe_advance_nonremovable_xid(RetainConflictInfoData *data, >bool status_received) > { > /* Exit early if retaining conflict information is not required */ > if (!MySubscription->retainconfl

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Tue, May 20, 2025 at 6:30 PM shveta malik wrote: > > Few more comments mostly for patch001: Thanks for the comments! > > 4) > For this feature, since we are only interested in remote UPDATEs happening > concurrently, so shall we ask primary to provide oldest "UPDATE" > transaction-id in comm

RE: Conflict detection for update_deleted in logical replication

2025-05-21 Thread Zhijie Hou (Fujitsu)
On Tue, May 20, 2025 at 11:08 AM shveta malik wrote: > > Please find few more comments: Thanks for the comments! > > 2) > > -- > send_feedback(last_received, requestReply, requestReply); > > + maybe_advance_nonremovable_xid(&data, false); > + > /* > * Force reporting to ensure l

RE: Fix slot synchronization with two_phase decoding enabled

2025-05-08 Thread Zhijie Hou (Fujitsu)
On Thu, May 8, 2025 at 6:04 PM Zhijie Hou (Fujitsu) wrote: > > On Tue, May 6, 2025 at 7:22 PM Zhijie Hou (Fujitsu) wrote: > > > > > On Mon, May 5, 2025 at 6:59 PM Amit Kapila wrote: > > > > > > On Sun, May 4, 2025 at 2:33 PM Masahiko Sawada > > &

RE: Fix slot synchronization with two_phase decoding enabled

2025-05-08 Thread Zhijie Hou (Fujitsu)
On Tue, May 6, 2025 at 7:22 PM Zhijie Hou (Fujitsu) wrote: > > On Mon, May 5, 2025 at 6:59 PM Amit Kapila wrote: > > > > On Sun, May 4, 2025 at 2:33 PM Masahiko Sawada > > > wrote: > > > > > > While I cannot be entirely certain of my analysis, I beli

RE: Fix slot synchronization with two_phase decoding enabled

2025-05-06 Thread Zhijie Hou (Fujitsu)
On Mon, May 5, 2025 at 6:59 PM Amit Kapila wrote: > > On Sun, May 4, 2025 at 2:33 PM Masahiko Sawada > wrote: > > > > While I cannot be entirely certain of my analysis, I believe the root > > cause might be related to the backward movement of the confirmed_flush > > LSN. The following scenario se

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-30 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 29, 2025 at 6:57 PM Amit Kapila wrote: > > On Tue, Apr 29, 2025 at 3:01 AM Masahiko Sawada > wrote: > > > > Your fix looks good to me. Is it worth considering putting an > > assertion to verify if new two_phase_at is equal to or greater than > > confirmed_lsn (or at least it doesn't g

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-28 Thread Zhijie Hou (Fujitsu)
On Mon, Apr 28, 2025 at 7:33 PM Zhijie Hou (Fujitsu) wrote: > > Thanks for reviewing. Here is V3 patch that addressed it. > > BTW, I also did some tests to confirm the catalog_xmin could still be > ahead in some case, and here is an example: > > 1. Create a failover r

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-28 Thread Zhijie Hou (Fujitsu)
On Mon, Apr 28, 2025 at 5:33 PM shveta malik wrote: > > On Mon, Apr 28, 2025 at 2:27 PM Zhijie Hou (Fujitsu) > wrote: > > > > On Fri, Apr 18, 2025 at 12:29 PM Amit Kapila wrote: > > > > > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) &

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-28 Thread Zhijie Hou (Fujitsu)
On Fri, Apr 18, 2025 at 12:29 PM Amit Kapila wrote: > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) > > > > - > > Fix > > - > > > > I think we should keep the confirmed_flush even if the previous synced > > restart_lsn/catalog_xm

RE: Fix premature xmin advancement during fast forward decoding

2025-04-27 Thread Zhijie Hou (Fujitsu)
On Sun, Apr 27, 2025 at 2:07 PM Masahiko Sawada wrote: > > On Sat, Apr 26, 2025 at 5:01 AM Amit Kapila > wrote: > > > > On Sat, Apr 26, 2025 at 12:01 AM Masahiko Sawada > wrote: > > > > > > On Fri, Apr 25, 2025 at 4:42 AM Amit Kapila > wrote: > > > > > > > > Can you think of any better ideas? >

RE: Fix premature xmin advancement during fast forward decoding

2025-04-24 Thread Zhijie Hou (Fujitsu)
On Fri, Apr 25, 2025 at 5:44 AM Masahiko Sawada wrote: (Resending this email after compressing the attachment) > On Tue, Apr 22, 2025 at 12:06 AM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > When analyzing some issues in another thread[1], I found a bug that &

RE: Fix premature xmin advancement during fast forward decoding

2025-04-23 Thread Zhijie Hou (Fujitsu)
On Wed, Apr 23, 2025 at 2:31 PM shveta malik wrote: > > On Tue, Apr 22, 2025 at 12:36 PM Zhijie Hou (Fujitsu) > wrote: > > > > Hi, > > > > To fix this, I think we can allow the base snapshot to be built during fast > > forward decoding, as implemented i

Fix premature xmin advancement during fast forward decoding

2025-04-22 Thread Zhijie Hou (Fujitsu)
Hi, When analyzing some issues in another thread[1], I found a bug that fast forward decoding could lead to premature advancement of catalog_xmin, resulting in required catalog data being removed by vacuum. The issue arises because we do not build a base snapshot when decoding changes during fast

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-21 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 22, 2025 at 11:23 AM Amit Kapila wrote: > > On Fri, Apr 18, 2025 at 9:58 AM Amit Kapila wrote: > > > > On Thu, Apr 17, 2025 at 6:14 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > - > > > Fix > > > - > > >

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-20 Thread Zhijie Hou (Fujitsu)
On Sat, Apr 19, 2025 at 2:19 AM Masahiko Sawada wrote: > > On Tue, Apr 8, 2025 at 10:14 PM Zhijie Hou (Fujitsu) > wrote: > > > > > > -- > > Approach 2 > > -- > > > > Instead of disallowing the use of two-phase and failover t

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 17, 2025 at 8:44 PM Zhijie Hou (Fujitsu) wrote: > > Hi, > > The recently added tests for slotsync on two-phase enabled slots > failed[1] due to a broken assertion triggered while decoding the > COMMIT PREPARED record on the promoted standby. Here is the missing

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-17 Thread Zhijie Hou (Fujitsu)
Hi, The recently added tests for slotsync on two-phase enabled slots failed[1] due to a broken assertion triggered while decoding the COMMIT PREPARED record on the promoted standby. - Analysis - if ((txn->final_lsn < two_phase_at) && is_commit) { /*

RE: Skipping schema changes in publication

2025-04-15 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 10, 2025 at 7:25 PM Amit Kapila wrote: > > On Tue, Jan 9, 2024 at 12:02 PM vignesh C wrote: > > > > As I did not see much interest from others, I'm withdrawing this patch > > for now. But if there is any interest others in future, I would be > > more than happy to work on this feature

Fix replica identity checks for MERGE command on published table.

2025-04-10 Thread Zhijie Hou (Fujitsu)
Hi, While testing publication DDLs, I noticed an unexpected behavior where the MERGE command can be executed on tables lacking replica identity keys, regardless of whether they are part of a publication that publishes updates and deletes. Replication and application of the updates and deletes gen

RE: Fix slot synchronization with two_phase decoding enabled

2025-04-08 Thread Zhijie Hou (Fujitsu)
On Thu, Apr 3, 2025 at 3:16 PM Zhijie Hou (Fujitsu) wrote: > > On Thu, Apr 3, 2025 at 1:38 PM Masahiko Sawada wrote: > > > > > On Wed, Apr 2, 2025 at 7:58 PM Amit Kapila > > wrote: > > > > > > On Thu, Apr 3, 2025 at 7:50 AM Zhijie Hou (Fujitsu) >

RE: BUG #18815: Logical replication worker Segmentation fault

2025-04-07 Thread Zhijie Hou (Fujitsu)
On Tue, Apr 8, 2025 at 2:00 PM Amit Kapila wrote: > > On Mon, Apr 7, 2025 at 3:28 PM Zhijie Hou (Fujitsu) > wrote: > > > > > What is the > > > behavior of conflict reporting code in case of exclusion constraints? > > > > Under logical replication cont

RE: BUG #18815: Logical replication worker Segmentation fault

2025-04-07 Thread Zhijie Hou (Fujitsu)
On Mon, Apr 7, 2025 at 5:37 PM Amit Kapila wrote: > > On Wed, Mar 26, 2025 at 10:3 AM Zhijie Hou (Fujitsu) > wrote: > > > > On Wed, Feb 19, 2025 at 4:38 AM Tom Lane wrote: > > > > My colleague Nisha reported off-list about a crash in logical replication

RE: pg_recvlogical cannot create slots with failover=true

2025-04-06 Thread Zhijie Hou (Fujitsu)
On Sat, Apr 5, 2025 at 1:45 AM Masahiko Sawada wrote: Hi, > Thank you for updating the patch! Pushed with small cosmetic changes. Thanks for pushing the feature ! I noticed one typo in the doc and here is a tiny patch to fix it. -The --two-phase and --falover options +The --two

  1   2   3   4   5   >