Re: Reduce useless changes before reassembly during logical replication

2024-03-06 Thread li jie
Hi, Sorry I replied too late. > This is all true but note that in successful cases (where the table is > published) all the work done by FilterByTable(accessing caches, > transaction-related stuff) can add noticeable overhead as anyway we do > that later in pgoutput_change(). You are correct.

Re: Reduce useless changes before reassembly during logical replication

2024-02-22 Thread Andy Fan
Hi, > > This is all true but note that in successful cases (where the table is > published) all the work done by FilterByTable(accessing caches, > transaction-related stuff) can add noticeable overhead as anyway we do > that later in pgoutput_change(). I think I gave the same comment > earlier

Re: Reduce useless changes before reassembly during logical replication

2024-02-21 Thread Andy Fan
Hi Jie, > Most importantly, if we filter out unpublished relationship-related > changes after > constructing the changes but before queuing the changes into a transaction, > will it reduce the workload of logical decoding and avoid disk or memory > growth > as much as possible? Thanks for the

Re: Reduce useless changes before reassembly during logical replication

2024-01-18 Thread Bharath Rupireddy
On Thu, Jan 18, 2024 at 2:47 PM Amit Kapila wrote: > > On Thu, Jan 18, 2024 at 12:12 PM Bharath Rupireddy > wrote: > > > > On Wed, Jan 17, 2024 at 11:45 AM li jie wrote: > > > > > > Hi hackers, > > > > > > During logical replication, if there is a large write transaction, some > > > spill files

Re: Reduce useless changes before reassembly during logical replication

2024-01-18 Thread Amit Kapila
On Thu, Jan 18, 2024 at 12:12 PM Bharath Rupireddy wrote: > > On Wed, Jan 17, 2024 at 11:45 AM li jie wrote: > > > > Hi hackers, > > > > During logical replication, if there is a large write transaction, some > > spill files will be written to disk, depending on the setting of > >

Re: Reduce useless changes before reassembly during logical replication

2024-01-17 Thread Bharath Rupireddy
On Wed, Jan 17, 2024 at 11:45 AM li jie wrote: > > Hi hackers, > > During logical replication, if there is a large write transaction, some > spill files will be written to disk, depending on the setting of > logical_decoding_work_mem. > > This behavior can effectively avoid OOM, but if the

Reduce useless changes before reassembly during logical replication

2024-01-16 Thread li jie
Hi hackers, During logical replication, if there is a large write transaction, some spill files will be written to disk, depending on the setting of logical_decoding_work_mem. This behavior can effectively avoid OOM, but if the transaction generates a lot of change before commit, a large number

Reduce useless changes before reassembly during logical replication

2023-12-07 Thread li jie
Hi hackers, During logical replication, if there is a large write transaction, some spill files will be written to disk, depending on the setting of logical_decoding_work_mem. This behavior can effectively avoid OOM, but if the transaction generates a lot of change before commit, a large number