RE: Failed transaction statistics to measure the logical replication progress

On Mon, Feb 21, 2022 11:46 AM Osumi, Takamichi/大墨 昂道 wrote: >I've addressed this point in a new v23 patch, since there was no opinion on >this so far. >Kindly have a look at the attached one. Thanks for updating the patch. Here is a comment: In function apply_handle_stream_abort: @@ -1217,6 +12

RE: Logical replication timeout problem

On Fri, Feb 18, 2022 at 10:51 AM Ajin Cherian wrote: > Some comments: Thanks for your review. > I see you only track skipped Inserts/Updates and Deletes. What about > DDL operations that are skipped, what about truncate. > What about changes made to unpublished tables? I wonder if you could > cr

RE: logical replication empty transactions

On Feb, Wed 23, 2022 at 10:58 PM Ajin Cherian wrote: > Few comments to V19-0001: 1. I think we should adjust the alignment format. git am ../v19-0001-Skip-empty-transactions-for-logical-replication.patch .git/rebase-apply/patch:197: indent with spaces. * Before we send schema, make sure that

RE: Logical replication timeout problem

On Wed, Feb 22, 2022 at 4:56 PM Amit Kapila wrote: > Thanks for your review. > On Tue, Feb 22, 2022 at 9:17 AM wangw.f...@fujitsu.com > wrote: > > > > On Fri, Feb 18, 2022 at 10:51 AM Ajin Cherian wrote: > > > Some comments: > > Thanks for your review. >

RE: Logical replication timeout problem

On Thur, Feb 24, 2022 at 4:06 PM Kuroda, Hayato/黒田 隼人 wrote: > Dear Wang, Thanks for your review. > > According to our discussion, we need to send keepalive messages to > > subscriber when skipping changes. > > One approach is that **for each skipped change**, we try to send > > keepalive messag

RE: Logical replication timeout problem

On Mon, Feb 28, 2022 at 6:58 PM Kuroda, Hayato/黒田 隼人 wrote: > Dear Wang, > > > Attached a new patch that addresses following improvements I have got > > so far as > > comments: > > 1. Consider other changes that need to be skipped(truncate, DDL and > > function calls pg_logical_emit_message). [s

RE: Logical replication timeout problem

On Fri, Mar 4, 2022 at 4:26 PM Kuroda, Hayato/黒田 隼人 wrote: > Thanks for your test and comments. > Some codes were added in ReorderBufferProcessTXN() for treating DDL, > but I doubted updating accept_writes is needed. > IMU, the parameter is read by OutputPluginPrepareWrite() in order align > mes

RE: Logical replication timeout problem

On Tues, Mar 8, 2022 at 11:54 PM Ajin Cherian wrote: > Some comments. Thanks for your comments. > 1. The comment on top of SendKeepaliveIfNecessary > > Try to send a keepalive message if too many changes was skipped. > > change to > > Try to send a keepalive message if too many changes wer s

RE: Logical replication timeout problem

On Tue, Mar 8, 2022 at 3:52 PM Masahiko Sawada wrote: > I've looked at the patch and have a question: Thanks for your review and comments. > +void > +SendKeepaliveIfNecessary(LogicalDecodingContext *ctx, bool skipped) { > +static int skipped_changes_count = 0; > + > +/* > +

RE: Logical replication timeout problem

On Tue, Mar 8, 2022 at 4:48 PM Kuroda, Hayato/黒田 隼人 wrote: > Thank you for updating the patch! Good self-reviewing. Thanks for your comments. > > And I looked into the function WalSndUpdateProgress. I found function > > WalSndUpdateProgress try to record the time of some message(by > > function

RE: Column Filtering in Logical Replication

On Fri, Mar 11, 2022 at 9:57 AM Tomas Vondra wrote: > Hi Tomas, Thanks for your patches. On Mon, Mar 9, 2022 at 9:53 PM Tomas Vondra wrote: >On Wed, Mar 9, 2022 at 6:04 PM Amit Kapila wrote: >>On Mon, Mar 7, 2022 at 11:18 PM Tomas Vondra >>wrote: >>>On Fri, Mar 4, 2022 at 6:43 PM Amit Kapil

RE: Logical replication timeout problem

On Wed, Mar 9, 2022 at 2:45 PM Masahiko Sawada wrote: > Thanks for your comments. > On Wed, Mar 9, 2022 at 10:26 AM I wrote: > > On Tue, Mar 8, 2022 at 3:52 PM Masahiko Sawada > wrote: > > > I've looked at the patch and have a question: > > Thanks for your review and comments. > > > > > +void >

RE: Logical replication timeout problem

On Thu, Mar 17, 2022 at 7:52 PM Masahiko Sawada wrote: > Thanks for your comments. > On Thu, Mar 17, 2022 at 7:14 PM Amit Kapila wrote: > > > > On Thu, Mar 17, 2022 at 12:27 PM Amit Kapila > wrote: > > > > > > On Wed, Mar 16, 2022 at 7:38 PM Masahiko Sawada > wrote: > > > > > > > > After more

RE: Logical replication timeout problem

On Thu, Mar 9, 2022 at 11:52 AM Kuroda, Hayato/黒田 隼人 wrote: > Thank you for updating! Thanks for your comments. > 1. pgoutput_change > ``` > + bool is_send = true; > ``` > > My first impression is that is_send should be initialized to false, and it > will change > to true when OutputPlug

RE: Logical replication timeout problem

On Mon, Mar 21, 2022 at 1:31 PM Amit Kapila wrote: > Thanks for your comments. > On Fri, Mar 18, 2022 at 4:20 PM Amit Kapila wrote: > > > > On Fri, Mar 18, 2022 at 10:43 AM wangw.f...@fujitsu.com > > wrote: > > > > > > On Thu, Mar 17

Confused comment about drop replica identity index

Hi hackers, When I doing development based by PG, I found the following comment have a little problem in file src/include/catalog/pg_class.h. /* * an explicitly chosen candidate key's columns are used as replica identity. * Note this will still be set if the index has been dropped; in that case

RE: Confused comment about drop replica identity index

On Tue, Dec 15, 2021 at 11:25AM, Michael Paquier wrote: > Yeah, the comment is wrong. If the index of a REPLICA_IDENTITY_INDEX is > dropped, I recall that the behavior is the same as REPLICA_IDENTITY_NOTHING. Thank you for your response. I agreed that the comment is wrong. > Not sure about the

RE: Confused comment about drop replica identity index

On Tue, Dec 16, 2021 at 06:40AM, Michael Paquier wrote: > Would you like to write a patch to address all that? OK, I will push it soon. Regards, Wang wei

RE: Confused comment about drop replica identity index

On Tue, Dec 16, 2021 at 10:27AM, Michael Paquier wrote: > On Tue, Dec 16, 2021 at 06:40AM, Michael Paquier wrote: > > Would you like to write a patch to address all that? > > OK, I will push it soon. Here is a patch to correct wrong comment about REPLICA_IDENTITY_INDEX, And improve the pg-doc.

RE: Confused comment about drop replica identity index

On Tue, Dec 20, 2021 at 19:11PM, Michael Paquier wrote: > That's mostly fine. I have made some adjustments as per the attached. Thanks for reviewing. > + The default for non-system tables. Records the old values of the > columns > + of the primary key, if any. The default for n

RE: Failed transaction statistics to measure the logical replication progress

On Mon, Dec 22, 2021 at 6:14 PM osumi.takami...@fujitsu.com wrote: > > Attached the new patch v19. > I have a question on the v19-0002 patch: When I tested for this patch, I found pg_stat_subscription_workers has some unexpected data. For example: [Publisher] create table replica_test1(a int,

RE: Failed transaction statistics to measure the logical replication progress

On Wednesday, December 22, 2021 10:30 PM osumi.takami...@fujitsu.com wrote: > On Wednesday, December 22, 2021 8:38 PM I wrote: > > Do we expect these commit counts which come from empty transactions ? > This is another issue discussed in [1] > where the patch in the thread is a work in progress, I

RE: Optionally automatically disable logical replication subscriptions on error

On Thursday, December 16, 2021 8:51 PM osumi.takami...@fujitsu.com wrote: > Attached the updated patch v14. A comment to the timing of printing a log: After the log[1] was printed, I altered subscription's option (DISABLE_ON_ERROR) from true to false before invoking DisableSubscriptionOnError to

RE: row filtering for logical replication

On Mon, Dec 28, 2021 9:03 PM houzj.f...@fujitsu.com wrote: > Attach a top up patch 0004 which did the above changes. A few comments about v55-0001 and v55-0002. v55-0001 1. There is a typo at the last sentence of function(rowfilter_walker)'s comment. * (b) a user-defined function can be used

RE: row filtering for logical replication

On Thu, Jan 4, 2022 at 00:54 PM Peter Smith wrote: > Modified in v58 [1] as suggested Thanks for updating the patches. A few comments about v58-0001 and v58-0002. v58-0001 1. How about modifying the following loop in copy_table by using for_each_from instead of foreach? Like the invocation of for

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Sun, Feb 19, 2023 at 21:06 PM Wang, Wei/王 威 wrote: > On Thur, Feb 14, 2023 at 2:03 AM Andres Freund wrote: > > On 2023-02-13 14:06:57 +0530, Amit Kapila wrote: > > > > > The patch calls update_progress in change_cb_wrapper and other > > > > > wrappers which will miss the case of DDLs that gene

RE: Fix the description of GUC "max_locks_per_transaction" and "max_pred_locks_per_transaction" in guc_table.c

On Wed, Feb 22, 2023 at 8:37 AM Nathan Bossart wrote: > On Wed, Feb 15, 2023 at 08:16:43AM +0000, wangw.f...@fujitsu.com wrote: > > When I refer to the GUC "max_locks_per_transaction", I find that the > description > > of the shared lock table size in pg-doc[1] i

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Thur, Feb 23, 2023 at 18:41 PM Kuroda, Hayato/�\田 隼人 wrote: > Dear Wang, > > Thank you for making the patch. IIUC your patch basically can achieve that > output plugin > does not have to call UpdateProgress. Thanks for your review and comments. > I think the basic approach is as follows, is

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Tues, Feb 28, 2023 at 9:12 AM Peter Smith wrote: > Here are some comments for the v2-0001 patch. > > (I haven't looked at the v3 that was posted overnight; maybe some of > my comments have already been addressed.) Thanks for your comments. > == > General > > 1. (Info from the commit mes

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Tues, Feb 28, 2023 at 11:31 AM Osumi, Takamichi/大墨 昂道 wrote: > Hi, > > > On Monday, February 27, 2023 6:30 PM wangw.f...@fujitsu.com > wrote: > > Attach the new patch. > Thanks for sharing v3. Minor review comments and question. Thanks

RE: Logical replication timeout problem

On Thu, Mar 2, 2023 at 4:19 AM Gregory Stark (as CFM) wrote: > On Wed, 8 Feb 2023 at 15:04, Andres Freund wrote: > > > > Attached is a current, quite rough, prototype. It addresses some of the > > points > > raised, but far from all. There's also several XXXs/FIXMEs in it. I changed > >

RE: Support logical replication of DDLs

On Mon, Mar 6, 2023 14:34 AM Ajin Cherian wrote: > Changes are in patch 1 and patch 2 Thanks for updating the patch set. Here are some comments: For v-75-0002* patch. 1. In the function deparse_AlterRelation. + if ((sub->address.objectId != relId && +sub->a

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Fri, Mar 3, 2023 8:18 AM Peter Smith wrote: > Thanks for your comments. > 1. > + > +static void UpdateProgressAndKeepalive(LogicalDecodingContext *ctx, > +bool finished_xact); > + > +static bool is_keepalive_threshold_exceeded(LogicalDecodingContext *ctx); > > 1a. > There is an unnecess

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Tue, Mar 7, 2023 15:55 PM Kuroda, Hayato/黒田 隼人 wrote: > Dear Wang, > > Thank you for updating the patch! Followings are my comments. Thanks for your comments. > --- > 01. missing comments > You might miss the comment from Peter[1]. Or could you pin the related one? Since I think the functi

RE: Support logical replication of DDLs

On Mon, Mar 6, 2023 18:17 PM Wang, Wei/王 威 wrote: > On Mon, Mar 6, 2023 14:34 AM Ajin Cherian wrote: > > Changes are in patch 1 and patch 2 > > Thanks for updating the patch set. > > Here are some comments: Here are some more comments for v-75-0002* patch: 1. In the function deparse_AlterRela

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Mon, Mar 10, 2023 11:56 AM Amit Kapila wrote: > On Wed, Mar 8, 2023 at 8:24 AM wangw.f...@fujitsu.com > wrote: > > > > Attach the new patch. > > > > I think this combines multiple improvements in one patch. We can > consider all of them together or maybe it w

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Wed, Mar 8, 2023 19:06 PM Kuroda, Hayato/黒田 隼人 wrote: > Dear Wang, Thanks for your testing and comments. > --- > ``` > +/* > + * Update progress tracking and send keep alive (if required). > + */ > +static void > +UpdateProgressAndKeepalive(LogicalDecodingContext *ctx, bool finished_xact) >

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Wed, Mar 8, 2023 23:55 PM Osumi, Takamichi/大墨 昂道 wrote: > Hi, > > > On Wednesday, March 8, 2023 11:54 AM From: wangw.f...@fujitsu.com > wrote: > > Attach the new patch. > Thanks for sharing v6 ! Few minor comments for the same. Thanks for your comments. > (1)

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Thur, Mar 9, 2023 13:26 PM Peter Smith wrote: > Here are some review comments for v6-0001 Thanks for your comments. > == > General. > > 1. > There are lots of new comments saying: > /* don't call update progress, we didn't really make any */ > > but is the wording "call update progress"

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Mon, Mar 10, 2023 14:35 PM Amit Kapila wrote: > On Fri, Mar 10, 2023 at 11:17 AM Peter Smith wrote: > > > > On Fri, Mar 10, 2023 at 3:32 PM Amit Kapila wrote: > > > > > > On Thu, Mar 9, 2023 at 10:56 AM Peter Smith > wrote: > > > > > > > > 2. rollback_prepared_cb_wrapper > > > > > > > > /*

RE: Rework LogicalOutputPluginWriterUpdateProgress

On Fri, Mar 10, 2023 20:17 PM Osumi, Takamichi/大墨 昂道 wrote: > Hi, > > > On Friday, March 10, 2023 6:32 PM Wang, Wei/王 威 > wrote: > > Attach the new patch set. > Thanks for updating the patch ! One review comment on v7-0005. Thanks for your comment. > stream_start_cb_wrapper and stream_stop_c

RE: Support logical replication of DDLs

On Tues, Mar 14, 2023 12:17 PM Ajin Cherian wrote: > On Mon, Mar 13, 2023 at 2:24 AM Zheng Li wrote: > > > > Thanks for working on the test coverage for CREATE and ALTER TABLE. > > I've made fixes for some of the failures in the v79 patch set (0002, > > 0003 and 0004 are updated). The changes inc

RE: Data is copied twice when specifying both child and parent table in publication

On Thu, Mar 16, 2023 at 20:25 PM Amit Kapila wrote: > Thanks for your comments. > + if (server_version >= 16) > + { > + appendStringInfo(&cmd, "SELECT DISTINCT N.nspname, C.relname,\n" > + " ( SELECT array_agg(a.attname ORDER BY a.attnum)\n" > + "FROM pg_attribut

RE: Data is copied twice when specifying both child and parent table in publication

On Fri, Mar 17, 2023 at 20:07 PM Amit Kapila wrote: > On Fri, Mar 17, 2023 at 11:58 AM wangw.f...@fujitsu.com > wrote: > > > > On Thu, Mar 16, 2023 at 20:25 PM Amit Kapila > wrote: > > > > > > > Thanks for your comments

RE: Data is copied twice when specifying both child and parent table in publication

On Mon, Mar 20, 2023 at 18:15 PM Amit Kapila wrote: > Thanks for your comments. > On Mon, Mar 20, 2023 at 1:02 PM Peter Smith > wrote: > > > > > > == > > src/include/catalog/pg_proc.dat > > > > 4. > > +{ oid => '6119', > > + descr => 'get information of the tables in the given publication

RE: Data is copied twice when specifying both child and parent table in publication

On Sat, Mar 18, 2023 at 7:37 AM Jacob Champion wrote: > On Thu, Mar 16, 2023 at 11:28 PM wangw.f...@fujitsu.com > wrote: > > Attach the new patch set. Thanks for your comments and testing. > For example, the corner case mentioned in 0003, with multiple > publications

RE: Data is copied twice when specifying both child and parent table in publication

On Mon, Mar 20, 2023 at 15:32 PM Peter Smith wrote: > Here are some review comments for v17-0001. Thanks for your comments. > == > src/backend/catalog/pg_publication.c > > 1. filter_partitions > > -static List * > -filter_partitions(List *relids) > +static void > +filter_partitions(List *t

RE: Data is copied twice when specifying both child and parent table in publication

On Mon, Mar 20, 2023 at 21:18 PM Kuroda, Hayato/黒田 隼人 wrote: > Dear Wang, > > I have tested about multilevel partitions, and it worked well. > Followings are my comments for v18-0001. Thanks for your comments and testing. > 01. pg_get_publication_tables > > ``` > + ListCell *l

RE: Data is copied twice when specifying both child and parent table in publication

On Wed, Mar 22, 2023 at 12:50 PM Peter Smith wrote: > Here are some review comments for patch code of HEAD_v19-0001 Thanks for your comments. > == > doc/src/sgml/ref/create_publication.sgml > > 1. > + > + There can be a case where a subscription combines multiple > +

RE: Data is copied twice when specifying both child and parent table in publication

On Wed, Mar 22, 2023 at 14:32 PM Kuroda, Hayato/黒田 隼人 wrote: > Dear Wang, > > Thank you for updating patch! Following are comments form v19-0001. Thanks for your comments. > 01. logical-replication.sgml > > I found a following statement in logical-replication.sgml. I think this may > cause >

RE: Data is copied twice when specifying both child and parent table in publication

On Thu, Mar 23, 2023 at 12:27 PM Peter Smith wrote: > Here are some review comments for patch v20-0001. Thanks for your comments. > == > src/backend/commands/subscriptioncmds.c > > 3. fetch_table_list > > + /* Get the list of tables from the publisher. */ > + if (server_version >= 16)

RE: Data is copied twice when specifying both child and parent table in publication

On Fri, Mar 24, 2023 at 10:14 AM Peter Smith wrote: > Thanks for the information. > BTW, since this patch changes the signature of the API > pg_get_publication_tables, I assume the example in the CREATE > SUBSCRIPTION Notes [1] may not work anymore. The use case you mentioned is still work. >

RE: Data is copied twice when specifying both child and parent table in publication

On Fri, Mar 24, 2023 at 14:17 PM Amit Kapila wrote: > > > > == > > > > src/test/subscription/t/028_row_filter.pl > > > > > > > > 7. > > > > +# insert data into partitioned table. > > > > +$node_publisher->safe_psql('postgres', > > > > + "INSERT INTO tab_rowfilter_viaroot_part(a) VALUES(13), (1

RE: Data is copied twice when specifying both child and parent table in publication

On Fri, Mar 24, 2023 at 9:49 AM Peter Smith wrote: > On Thu, Mar 23, 2023 at 8:11 PM wangw.f...@fujitsu.com > wrote: > > > > On Thu, Mar 23, 2023 at 12:27 PM Peter Smith > wrote: > > > Here are some review comments for patch v20-0001. > > > ... >

RE: Perform streaming logical transactions by background workers and parallel apply

On Mon, Dec 26, 2022 21:02 PM Masahiko Sawada wrote: > Thank you for updating the patches. Here are some comments for 0001 > and 0002 patches: Thanks for your comments. > I think it'd be better to write logs when the leader enters the > serialization mode. It would be helpful for investigating i

RE: Logical replication timeout problem

On Fri, Jan 6, 2023 at 15:06 PM Ashutosh Bapat wrote: > Hi Wang, > Thanks for working on this. One of our customer faced a similar > situation when running BDR with PostgreSQL. > > I tested your patch and it solves the problem. > > Please find some review comments below Thanks for your testing

RE: Logical replication timeout problem

On Mon, Jan 9, 2023 at 13:04 PM Amit Kapila wrote: > Thanks for your comments. > One more thing, I think it would be better to expose a new callback > API via reorder buffer as suggested previously [2] similar to other > reorder buffer APIs instead of directly using reorderbuffer API to > invoke

Adjust the description of OutputPluginCallbacks in pg-doc

Hi, When I was reading the "Logical Decoding Output Plugins" chapter in pg-doc [1], I think in the summary section, only the callback message_cb is not described whether it is required or optional, and the description of callback stream_prepare_cb seems inaccurate. And after the summary section,

RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

On Wed, Jan 11, 2023 4:31 PM Melih Mutlu wrote: > Rebased the patch to resolve conflicts. Thanks for your patch set. Here are some comments: v3-0001* patch === 1. typedefs.list I think we also need to add "walrcv_slot_snapshot_fn" to this file. v7-0002* patch === 1. Ab

RE: Perform streaming logical transactions by background workers and parallel apply

On Wed, Jan 18, 2023 12:36 PM Amit Kapila wrote: > On Tue, Jan 17, 2023 at 8:07 PM Masahiko Sawada > wrote: > > > > On Tue, Jan 17, 2023 at 6:14 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Tuesday, January 17, 2023 2:46 PM Masahiko Sawada > wrote: > > > > > > > > On Tue, Jan 17, 2023 at

RE: Logical replication timeout problem

On Wed, Jan 18, 2023 at 13:29 PM Amit Kapila wrote: > On Tue, Jan 17, 2023 at 6:41 PM Ashutosh Bapat > wrote: > > > > On Tue, Jan 17, 2023 at 3:34 PM Amit Kapila wrote: > > > > > > > > > > I am a bit worried about the indirections that the wrappers and hooks > > > > create. Output plugins call O

RE: Data is copied twice when specifying both child and parent table in publication

On Mon, Mar 27, 2023 at 11:32 AM Amit Kapila wrote: > On Mon, Mar 27, 2023 at 7:03 AM Peter Smith > wrote: > > > > > > 1. > > +# two publications, one publishing through ancestor and another one > > directly > > +# publsihing the partition, with different row filters > > +$node_publisher->safe_p

RE: Data is copied twice when specifying both child and parent table in publication

On Tues, Mar 28, 2023 at 7:02 AM Jacob Champion wrote: > On Mon, Mar 20, 2023 at 11:22 PM Amit Kapila > wrote: > > If the tests you have in mind are only related to this patch set then > > feel free to propose them here if you feel the current ones are not > > sufficient. > > I think the new tes

RE: Data is copied twice when specifying both child and parent table in publication

On Tues, Mar 28, 2023 at 18:00 PM Wang, Wei/王 威 wrote: > Attach the new patch. Sorry, I attached the wrong patch. Here is the correct new version patch which addressed all comments so far. Regards, Wang Wei v25-0001-Avoid-syncing-data-twice-for-the-publish_via_par.patch Description: v25-0001-

RE: Fix the description of GUC "max_locks_per_transaction" and "max_pred_locks_per_transaction" in guc_table.c

On Tues, Apr 4, 2023 at 23:48 PM Tom Lane wrote: > Nathan Bossart writes: > > On Wed, Feb 22, 2023 at 12:40:07PM +, wangw.f...@fujitsu.com wrote: > >> After some rethinking, I think users can easily get exact value according > >> to > >> exact formula, an

RE: Logical replication timeout problem

On Thur, Mar 24, 2022 at 6:32 PM Amit Kapila wrote: > Thanks for your kindly update. > It seems by mistake you have removed the changes from pgoutput_message > and pgoutput_truncate functions. I have added those back. > Additionally, I made a few other changes: (a) moved the function > UpdateProg

RE: Logical replication timeout problem

On Fri, Mar 25, 2022 at 2:19 PM Masahiko Sawada wrote: > On Fri, Mar 25, 2022 at 2:23 PM wangw.f...@fujitsu.com > wrote: > > > > On Thur, Mar 24, 2022 at 6:32 PM Amit Kapila > wrote: > > > > > Thanks for your kindly update. > > > > > It

RE: Logical replication timeout problem

On Mon, Mar 28, 2022 at 9:56 AM Kuroda, Hayato/黒田 隼人 wrote: > Dear Wang-san, Thanks for your comments. > Thank you for updating! > ...but it also cannot be applied to current HEAD > because of the commit 923def9a533. > > Your patch seems to conflict the adding an argument of > logicalrep_write_

RE: Logical replication timeout problem

On Mon, Mar 28, 2022 at 2:11 AM I wrote: > Rebase the patch. After reviewing anohter patch[1], I think this patch should also add a loop in function WalSndUpdateProgress like what did in function WalSndWriteData. So update the patch to be consistent with the existing code and the patch mentioned a

RE: Logical replication timeout problem

On Tues, Mar 29, 2022 at 9:45 AM I wrote: > Attach the new patch. Rebase the patch because the commit d5a9d86d in current HEAD. Regards, Wang wei v10-0001-Fix-the-logical-replication-timeout-during-large.patch Description: v10-0001-Fix-the-logical-replication-timeout-during-large.patch

RE: Logical replication timeout problem

On Fri, Apr 1, 2022 at 12:09 AM Amit Kapila wrote: > On Fri, Apr 1, 2022 at 8:28 AM Euler Taveira wrote: > > > > On Thu, Mar 31, 2022, at 11:27 PM, Amit Kapila wrote: > > > > This is exactly our initial analysis and we have tried a patch on > > these lines and it has a noticeable overhead. See [1

RE: Logical replication timeout problem

On Wed, Apr 6, 2022 at 1:59 AM Amit Kapila wrote: On Wed, Apr 6, 2022 at 4:32 AM Amit Kapila wrote: > Thanks for your comments. > typedef void (*LogicalOutputPluginWriterUpdateProgress) (struct > LogicalDecodingContext *lr, > XLogRecPtr Ptr, > TransactionId xid, > - bool skipped_xact > + bo

RE: Data is copied twice when specifying both child and parent table in publication

On Thur, Mar 10, 2021 at 10:08 AM houzj.f...@fujitsu.com wrote: > Hi, > > When reviewing some logical replication related features. I noticed another > possible problem if the subscriber subscribes multiple publications which > publish parent and child table. > > For example: > > pub > creat

RE: Logical replication timeout problem

On Wed, Apr 7, 2022 at 1:34 PM Amit Kapila wrote: > Thanks for your comments. > One comment: > +static void > +update_progress(LogicalDecodingContext *ctx, bool skipped_xact, bool > end_xact) > +{ > + static int changes_count = 0; > + > + if (end_xact) > + { > + /* Update progress tracking at xa

RE: Logical replication timeout problem

On Wed, Apr 7, 2022 at 1:34 PM Amit Kapila wrote: > On Wed, Apr 6, 2022 at 6:30 PM wangw.f...@fujitsu.com > wrote: > > > > On Wed, Apr 6, 2022 at 1:58 AM Amit Kapila wrote: > > On Wed, Apr 6, 2022 at 4:32 AM Amit Kapila wrote: > > > Also, let's try to eva

RE: Logical replication timeout problem

On Mon, Apr 11, 2022 at 2:39 PM I wrote: > Attach the new patch. Also, share test results and details. To check that the lsn information used for the calculation is what we expected, I get some information by adding logs in the function LagTrackerRead. Summary of test results: - In current HEAD a

RE: Skipping schema changes in publication

On Tue, Apr 12, 2022 at 2:23 PM vignesh C wrote: > The patch does not apply on top of HEAD because of the recent commit, > attached patch is rebased on top of HEAD. Thanks for your patches. Here are some comments for v1-0001: 1. I found the patch add the following two new functions in gram.y: pre

RE: Logical replication timeout problem

> > > > > > On Mon, Apr 11, 2022 at 12:09 PM wangw.f...@fujitsu.com > > > > wrote: > > > > > > > > > > So I skip tracking lag during a transaction just like the current > > > > > HEAD. > > > > > At

RE: Logical replication timeout problem

On Thur, Apr 14, 2022 at 8:21 PM Euler Taveira wrote: > Thanks for your comments. > + * For a large transaction, if we don't send any change to the downstream > for a > + * long time then it can timeout. This can happen when all or most of the > + * changes are either not published or got filter

RE: Logical replication timeout problem

On Mon, Apr 19, 2022 at 9:32 AM Masahiko Sawada wrote: > Thank you for updating the patch. Thanks for your comments. > + * For a large transaction, if we don't send any change to the > + downstream for a > + * long time(exceeds the wal_receiver_timeout of standby) then it can > timeout. > + * Thi

RE: Logical replication timeout problem

On Mon, Apr 18, 2022 at 00:36 PM Amit Kapila wrote: > On Mon, Apr 18, 2022 at 9:29 AM Amit Kapila wrote: > > > > On Thu, Apr 14, 2022 at 5:52 PM Euler Taveira wrote: > > > > > > On Wed, Apr 13, 2022, at 7:45 AM, Amit Kapila wrote: > > > > > > Sawada-San, Euler, do you have any opinion on this ap

RE: Logical replication timeout problem

On Wed, Apr 20, 2022 at 6:13 PM Amit Kapila wrote: > On Wed, Apr 20, 2022 at 2:38 PM Amit Kapila wrote: > > > > On Wed, Apr 20, 2022 at 12:51 PM Masahiko Sawada > wrote: > > > > > > On Wed, Apr 20, 2022 at 11:46 AM wangw.f...@fujitsu.com > > > wrot

RE: Data is copied twice when specifying both child and parent table in publication

On Tue, Apr 19, 2022 4:53 PM Shi, Yu/侍 雨 wrote: > On Tue, Apr 19, 2022 3:05 PM houzj.f...@fujitsu.com > wrote: > > > > > -Original Message- > > > From: Wang, Wei/王 威 > > On Thursday, April 7, 2022 11:08 AM > > > > > > On Thur, Mar 10, 2021 at 10:08 AM houzj.f...@fujitsu.com wrote: > > >

RE: Logical replication timeout problem

On Wed, Apr 21, 2022 at 10:15 AM I wrote: > The comments by Sawada-San sound reasonable to me. > After doing check, I found that padding in HEAD is the same as in REL14. > So I change the approach of patch for HEAD just like the patch for REL14. Also attach the back-branch patches for REL10~REL13.

RE: Data is copied twice when specifying both child and parent table in publication

On Thur, Apr 21, 2022 at 5:41 PM Amit Kapila wrote: > Thanks for your comments. > On Tue, Apr 19, 2022 at 2:23 PM shiy.f...@fujitsu.com > wrote: > > > > On Tue, Apr 19, 2022 3:05 PM houzj.f...@fujitsu.com > wrote: > > > > > > One suggestion is that can we simplify the code by moving the logic >

RE: Data is copied twice when specifying both child and parent table in publication

On Sun, Apr 24, 2022 at 2:16 PM I wrote: > On Thur, Apr 21, 2022 at 5:41 PM Amit Kapila wrote: > > IIRC, the column list and row filter also have some issues exactly due to > > this > > reason, so, I would like those cases to be also mentioned here and probably > > include the tests for them in t

RE: Logical replication timeout problem

On Wed, Jan 19, 2022 at 9:53 PM Fabrice Chapuis fabrice636...@gmail.com wrote: > Hello Amit, > If it takes little work for you, can you please send me a piece of code > with the change needed to do the test I wrote a patch(Send-keepalive.patch, please refer to atta

RE: Logical replication timeout problem

On Thu, Jan 20, 2022 at 9:18 PM Amit Kapila wrote: > It might be not reaching the actual send_keep_alive logic in > WalSndKeepaliveIfNecessary because of below code: > { > ... > /* > * Don't send keepalive messages if timeouts are globally disabled or > * we're doing something not partaking in tim

RE: Logical replication timeout problem

On Thu, Jan 22, 2022 at 7:12 PM Amit Kapila wrote: > Now, one idea to solve this problem could be that whenever we skip > sending any change we do try to update the plugin progress via > OutputPluginUpdateProgress(for walsender, it will invoke > WalSndUpdateProgress), and there it tries to process

RE: Logical replication timeout problem

On Sat, Jan 28, 2022 at 19:36 PM Fabrice Chapuis wrote: > shouldn't we use receiver_timeout in place of wal_sender_timeout because de > problem comes from the consummer. Thanks for your review. IMO, because it is a bug fix on the publisher-side, and the keepalive message is sent based on wal_sen

RE: Logical replication timeout problem

On Wed, Jan 26, 2022 at 11:37 AM I wrote: > On Sat, Jan 22, 2022 at 7:12 PM Amit Kapila wrote: > > Now, one idea to solve this problem could be that whenever we skip > > sending any change we do try to update the plugin progress via > > OutputPluginUpdateProgress(for walsender, it will invoke > >

RE: Logical replication timeout problem

On Tues, Feb 08, 2022 at 17:18 PM Kuroda, Hayato wrote: > I applied your patch and confirmed that codes passed regression test. > I put a short reviewing: Thanks for your test and review. > ``` > + static int skipped_changes_count = 0; > + /* > + * Conservatively, at least 150,000 c

RE: Logical replication timeout problem

On Fri, Nov 4, 2022 at 18:13 PM Fabrice Chapuis wrote: > Hello Wang, > > I tested the draft patch in my lab for Postgres 14.4, the refresh of the > materialized view ran without generating the timeout on the worker. > Do you plan to propose this patch at the next commit fest. Thanks for your con

RE: Data is copied twice when specifying both child and parent table in publication

On Fri, Oct 21, 2022 at 17:02 PM Peter Smith wrote: > Thanks for your comments. Sorry for not replying in time. > On Mon, Oct 17, 2022 at 4:49 PM wangw.f...@fujitsu.com > wrote: > > > > On Wed, Oct 5, 2022 at 11:08 AM Peter Smith > wrote: > > > Hi Wang-san

RE: Data is copied twice when specifying both child and parent table in publication

On Fri, Oct 21, 2022 at 17:02 PM Peter Smith wrote: > Here are my review comments for HEAD patches v13* Thanks for your comments. > Patch HEAD_v13-0002 > > 1. Commit message > > The following usage scenarios are not described in detail in the manual: > If one subscription subscribes multiple

RE: Data is copied twice when specifying both child and parent table in publication

On Tues, Nov 8, 2022 at 12:12 PM Osumi, Takamichi/大墨 昂道 wrote: > On Monday, October 17, 2022 2:49 PM Wang, Wei/王 威 > wrote: > > Attach the new patch set. > Hi, thank you for posting the new patches. > > > Here are minor comments on the HEAD_v13-0002. Thanks for your comments. > (1) Suggestio

RE: Data is copied twice when specifying both child and parent table in publication

On Mon, Nov 14, 2022 at 0:56 AM vignesh C wrote: > > > > Attach new patches. > Thanks for your comments. > Here we are having tables list to store the relids and table_infos > list which stores pubid along with relid. Here tables list acts as a > temporary list to get filter_partitions and then

RE: Data is copied twice when specifying both child and parent table in publication

On Thurs, Nov 17, 2022 at 13:58 PM vignesh C wrote: > On Wed, 16 Nov 2022 at 14:28, wangw.f...@fujitsu.com > wrote: > > > > On Mon, Nov 14, 2022 at 0:56 AM vignesh C wrote: > > > > > > > > Attach new patches. > > > > > > > Thanks

RE: Avoid distributing new catalog snapshot again for the transaction being decoded.

On Sat, Nov 26, 2022 at 19:50 PM Amit Kapila wrote: > On Fri, Nov 25, 2022 at 5:30 PM Ashutosh Bapat > wrote: > > > > Hi Hou, > > Thanks for the patch. With a simple condition, we can eliminate the > > need to queueing snapshot change in the current transaction and then > > applying it. Saves som

An omission of automatic completion in tab-complete.c

There is an omission of automatic completion of CURRENT_ROLE in tab-complete.c. Best wishes, Wei Wang An-omission-of-automatic-completion-of-CURRENT_ROLE.patch Description: An-omission-of-automatic-completion-of-CURRENT_ROLE.patch

  1   2   >