RE: Table refer leak in logical replication

2021-04-05 Thread shiy.f...@fujitsu.com
> BTW, it seems better to add a testcase for this ? I think the test for it can be added in src/test/subscription/t/003_constraints.pl, which is like what in my patch. Regards, Shi yu tests_for_table_refer_leak.diff Description: tests_for_table_refer_leak.diff

RE: psql - add SHOW_ALL_RESULTS option

2021-04-07 Thread shiy.f...@fujitsu.com
Hi I met a problem after commit 3a51306722. While executing a SQL statement with psql, I can't interrupt it by pressing ctrl+c. For example: postgres=# insert into test select generate_series(1,1000); ^C^CINSERT 0 1000 Press ctrl+c before finishing INSERT, and psql still continuing

RE: psql - add SHOW_ALL_RESULTS option

2021-04-07 Thread shiy.f...@fujitsu.com
> Attached a patch which attempts to fix this by moving the cancellation > cancelling request after processing results. Thank you for your fixing. I tested and the problem has been solved after applying your patch. Regards, Shi yu

RE: Could you help testing logical replication?

2021-04-12 Thread shiy.f...@fujitsu.com
> Then I get timeout error occurs and the subscriber worker keep re-launching > over and over (you did not mention see such errors?) I test again and get errors, too. I didn't check log after timeout in the previous test. Regards, Tang

撤回: Could you help testing logical replication?

2021-04-12 Thread shiy.f...@fujitsu.com
Shi, Yu/侍 雨 将撤回邮件“Could you help testing logical replication?”。

撤回: Could you help testing logical replication?

2021-04-12 Thread shiy.f...@fujitsu.com
Shi, Yu/侍 雨 将撤回邮件“Could you help testing logical replication?”。

RE: Could you help testing logical replication?

2021-04-12 Thread shiy.f...@fujitsu.com
Sorry for sending a wrong mail. Please ignore it. > -Original Message- > From: Shi, Yu/侍 雨 > Sent: Monday, April 12, 2021 6:51 PM > To: Tang, Haiying/唐 海英 > Cc: pgsql-hackers@lists.postgresql.org > Subject: RE: Could you help testing logical replication? > > > Then I get timeout error

RE: Truncate in synchronous logical replication failed

2021-04-22 Thread shiy.f...@fujitsu.com
> I don't think here we need to restart to get a stable list of indexes > as we do in RelationGetIndexAttrBitmap. The reason is here we build > the cache entry using a historic snapshot and all the later changes > are absorbed while decoding WAL. I have updated that and modified few > comments in

RE: Data is copied twice when specifying both child and parent table in publication

2021-10-21 Thread shiy.f...@fujitsu.com
On Tuesday, October 19, 2021 10:47 AM houzj.f...@fujitsu.com wrote: > > On Monday, October 18, 2021 5:03 PM Amit Langote > wrote: > > I can imagine that the behavior seen here may look surprising, but not > > sure if I would call it a bug as such. I do remember thinking about > > this case

RE: logical replication empty transactions

2022-03-07 Thread shiy.f...@fujitsu.com
On Fri, Mar 4, 2022 9:41 AM Ajin Cherian wrote: > > I have split the patch into two. I have kept the logic of skipping > streaming changes in the second patch. > I will work on the second patch once we can figure out a solution for > the COMMIT PREPARED after restart problem. > Thanks for

RE: Optionally automatically disable logical replication subscriptions on error

2022-03-06 Thread shiy.f...@fujitsu.com
On Wed, Mar 2, 2022 5:39 PM osumi.takami...@fujitsu.com wrote: > > Attached an updated patch v26. > Thanks for your patch. A comment on the document. @@ -7771,6 +7771,16 @@ SCRAM-SHA-256$iteration count: + subdisableonerr bool + + + If true, the

RE: Skipping logical replication transactions on subscriber side

2022-03-14 Thread shiy.f...@fujitsu.com
On Fri, Mar 11, 2022 4:20 PM Masahiko Sawada wrote: > > I've attached an updated version patch. This patch can be applied on > top of the latest disable_on_error patch[1]. > Thanks for your patch. Here are some comments for the v13 patch. 1. doc/src/sgml/ref/alter_subscription.sgml +

RE: Skipping logical replication transactions on subscriber side

2022-03-16 Thread shiy.f...@fujitsu.com
On Wed, Mar 16, 2022 4:23 PM Masahiko Sawada wrote: > > I've attached an updated version patch. > Thanks for updating the patch. Here are some comments for the v15 patch. 1. src/backend/replication/logical/worker.c + * to skip applying the changes when starting to apply changes. The

RE: Column Filtering in Logical Replication

2022-03-14 Thread shiy.f...@fujitsu.com
On Mon, Mar 14, 2022 5:08 AM Tomas Vondra wrote: > > On 3/12/22 05:30, Amit Kapila wrote: > >> ... > > > > Okay, please find attached. I have done basic testing of this, if we > > agree with this approach then this will require some more testing. > > > > Thanks, the proposed changes seem like a

RE: logical replication empty transactions

2022-03-01 Thread shiy.f...@fujitsu.com
Hi, Here are some comments on the v21 patch. 1. + WalSndKeepalive(false, 0); Maybe we can use InvalidXLogRecPtr here, instead of 0. 2. + pq_sendint64(_message, writePtr ? writePtr : sentPtr); Similarly, should we use XLogRecPtrIsInvalid()? 3. @@ -1183,6 +1269,20

RE: Failed transaction statistics to measure the logical replication progress

2022-03-02 Thread shiy.f...@fujitsu.com
Hi, A comments on the v26 patch. The following document about pg_stat_subscription_stats view only says that "showing statistics about errors", should we add something about transactions here? pg_stat_subscription_statspg_stat_subscription_stats One row per subscription,

RE: row filtering for logical replication

2022-03-02 Thread shiy.f...@fujitsu.com
On Thu, Mar 3, 2022 10:40 AM Amit Kapila wrote: > > On Wed, Mar 2, 2022 at 5:42 PM Euler Taveira wrote: > > > > On Wed, Mar 2, 2022, at 8:45 AM, Tomas Vondra wrote: > > > > While working on the column filtering patch, which touches about the > > same places, I noticed two minor gaps in testing:

RE: logical replication empty transactions

2022-03-23 Thread shiy.f...@fujitsu.com
On Thursday, March 24, 2022 11:19 AM Hou, Zhijie/侯 志杰 wrote: > > Attach the new version patch which include the following changes: > > - Fix a typo > - Change the requestreply flag of the newly added WalSndKeepalive to false, > because the subscriber can judge whether it's necessary to post

RE: Logical replication timeout problem

2022-03-30 Thread shiy.f...@fujitsu.com
On Wed, Mar 30, 2022 3:54 PM wangw.f...@fujitsu.com wrote: > > Rebase the patch because the commit d5a9d86d in current HEAD. > Thanks for your patch, I tried this patch and confirmed that there is no timeout problem after applying this patch, and I could reproduce this problem on HEAD.

RE: logical replication empty transactions

2022-03-29 Thread shiy.f...@fujitsu.com
On Tue, Mar 29, 2022 5:15 PM Hou, Zhijie/侯 志杰 wrote: > > Thanks for the comment. > Attach the new version patch with this change. > Hi, I did a performance test for this patch to see if it affects performance when publishing empty transactions, based on the v32 patch. In this test, I use

RE: Skipping schema changes in publication

2022-04-14 Thread shiy.f...@fujitsu.com
On Tue, Apr 12, 2022 2:23 PM vignesh C wrote: > > The patch does not apply on top of HEAD because of the recent commit, > attached patch is rebased on top of HEAD. > Thanks for your patch. Here are some comments for 0001 patch. 1. doc/src/sgml/catalogs.sgml @@ -6438,6 +6438,15 @@

RE: Data is copied twice when specifying both child and parent table in publication

2022-04-19 Thread shiy.f...@fujitsu.com
On Tue, Apr 19, 2022 3:05 PM houzj.f...@fujitsu.com wrote: > > > -Original Message- > > From: Wang, Wei/王 威 > On Thursday, April 7, 2022 11:08 AM > > > > On Thur, Mar 10, 2021 at 10:08 AM houzj.f...@fujitsu.com wrote: > > > Hi, > > > > > > When reviewing some logical replication related

RE: bogus: logical replication rows/cols combinations

2022-05-16 Thread shiy.f...@fujitsu.com
On Mon, May 16, 2022 8:34 PM houzj.f...@fujitsu.com wrote: > > Attach the new version patch. > Thanks for your patch. Here are some comments: 1. (0001 patch) /* * Returns Oids of tables in a publication. */ Datum pg_get_publication_tables(PG_FUNCTION_ARGS) Should we modify the comment of

RE: Skipping schema changes in publication

2022-05-17 Thread shiy.f...@fujitsu.com
On Sat, May 14, 2022 9:33 PM vignesh C wrote: > > Thanks for the comments, the attached v5 patch has the changes for the > same. Also I have made the changes for SKIP Table based on the new > syntax, the changes for the same are available in >

RE: Handle infinite recursion in logical replication setup

2022-05-19 Thread shiy.f...@fujitsu.com
On Wed, May 4, 2022 2:47 PM vignesh C wrote: > > Thanks for the comments, the attached v13 patch has the changes for the same. > Here are some comments on v13-0002 patch. 1) +* Throw an error so that the user can take care of the initial data +* copying and

RE: Handle infinite recursion in logical replication setup

2022-05-23 Thread shiy.f...@fujitsu.com
On Tue, May 24, 2022 1:34 AM vignesh C wrote: > > Thanks for the comments, the attached v15 patch has the fixes for the same. > Thanks for updating the patch. I have a comment on the document in 0002 patch. @@ -300,6 +310,11 @@ CREATE SUBSCRIPTION subscription_namefalse. +

RE: Hash index build performance tweak from sorting

2022-05-30 Thread shiy.f...@fujitsu.com
On Tue, May 10, 2022 5:43 PM Simon Riggs wrote: > > On Sat, 30 Apr 2022 at 12:12, Amit Kapila > wrote: > > > > Few comments on the patch: > > 1. I think it is better to use DatumGetUInt32 to fetch the hash key as > > the nearby code is using. > > 2. You may want to change the below comment in

RE: Handle infinite recursion in logical replication setup

2022-05-27 Thread shiy.f...@fujitsu.com
On Wed, May 25, 2022 7:55 PM vignesh C wrote: > > The attached v16 patch has the changes for the same. > Thanks for updating the patch. Some comments for the document in 0002 patch. 1. + +Lock the required tables in node1 and +node2 till the setup is completed. + + + +

RE: Handle infinite recursion in logical replication setup

2022-05-18 Thread shiy.f...@fujitsu.com
On Wed, May 4, 2022 2:47 PM vignesh C wrote: > > Thanks for the comments, the attached v13 patch has the changes for the same. > Thanks for your patch. Here are some comments on v13-0001 patch. 1) @@ -152,6 +152,18 @@ CREATE SUBSCRIPTION subscription_name + +

RE: Handle infinite recursion in logical replication setup

2022-07-04 Thread shiy.f...@fujitsu.com
On Mon, Jul 4, 2022 4:17 PM shiy.f...@fujitsu.com wrote: > > On Sun, Jul 3, 2022 11:00 PM vignesh C wrote: > > > > Thanks for the comments, the attached v27 patch has the changes for the > > same. > > > > Thanks for updating the patch. > > A comment

RE: Handle infinite recursion in logical replication setup

2022-07-04 Thread shiy.f...@fujitsu.com
On Sun, Jul 3, 2022 11:00 PM vignesh C wrote: > > Thanks for the comments, the attached v27 patch has the changes for the > same. > Thanks for updating the patch. A comment on 0003 patch: + /* +* No need to throw an error for the tables that are in ready state,

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-11 Thread shiy.f...@fujitsu.com
On Tue, Jul 12, 2022 8:49 AM Masahiko Sawada wrote: > > I've attached an updated patch. > > While trying this idea, I noticed there is no API to get the length of > dlist, as we discussed offlist. Alternative idea was to use List > (T_XidList) but I'm not sure it's a great idea since deleting

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread shiy.f...@fujitsu.com
On Tue, Jul 12, 2022 8:49 AM Masahiko Sawada wrote: > > I've attached an updated patch. > Hi, I met a segmentation fault in test_decoding test after applying the patch for master branch. Attach the backtrace. It happened when executing the following code because it tried to free a NULL

RE: Handle infinite recursion in logical replication setup

2022-06-29 Thread shiy.f...@fujitsu.com
On Tue, Jun 28, 2022 2:18 PM vignesh C wrote: > > Thanks for the comments, the attached v25 patch has the changes for the > same. > Thanks for updating the patch. Here are some comments. 0002 patch: == 1. +# Test the CREATE SUBSCRIPTION 'origin' parameter and its interaction with

RE: Perform streaming logical transactions by background workers and parallel apply

2022-07-06 Thread shiy.f...@fujitsu.com
On Tue, Jun 28, 2022 11:22 AM Wang, Wei/王 威 wrote: > > I also improved patches as suggested by Peter-san in [1] and [2]. > Thanks for Shi Yu to improve the patches by addressing the comments in [2]. > > Attach the new patches. > Thanks for updating the patch. Here are some comments. 0001

RE: tablesync copy ignores publication actions

2022-06-22 Thread shiy.f...@fujitsu.com
On Wed, Jun 22, 2022 4:49 PM Peter Smith wrote: > > On Wed, Jun 22, 2022 at 2:18 PM Amit Kapila > wrote: > > > > On Thu, Jun 16, 2022 at 6:07 AM Peter Smith > wrote: > > > > > > > > Thank you for your review comments. Those reported mistakes are fixed > > > in the attached patch v3. > > > > >

RE: Handle infinite recursion in logical replication setup

2022-06-22 Thread shiy.f...@fujitsu.com
On Mon, Jun 20, 2022 7:55 PM vignesh C wrote: > > Thanks for the comment, the v22 patch attached has the changes for the > same. Thanks for updating the patch, here are some comments on 0003 patch. 1. 032_origin.pl

Replica Identity check of partition table on subscriber

2022-06-08 Thread shiy.f...@fujitsu.com
Hi hackers, I saw a problem in logical replication, when the target table on subscriber is a partitioned table, it only checks whether the Replica Identity of partitioned table is consistent with the publisher, and doesn't check Replica Identity of the partition. For example: -- publisher --

RE: Replica Identity check of partition table on subscriber

2022-06-10 Thread shiy.f...@fujitsu.com
On Thu, June 9, 2022 7:02 PM Amit Kapila wrote: > > > I think one approach to fix it is to check the target partition in this > > case, > > instead of the partitioned table. > > > > This approach sounds reasonable to me. One minor point: > +/* > + * Check that replica identity matches. > + * >

RE: Handle infinite recursion in logical replication setup

2022-06-06 Thread shiy.f...@fujitsu.com
On Mon, Jun 6, 2022 1:14 PM vignesh C wrote: > > The attached v18 patch has the fixes for the same. > Thanks for updating the patch, here are some comments. 0002 patch == 1. + +origin (string) + + It maybe better if the type of "origin" parameter

RE: tablesync copy ignores publication actions

2022-06-13 Thread shiy.f...@fujitsu.com
On Wed, Jun 8, 2022 12:10 PM Amit Kapila wrote: > > On Tue, Jun 7, 2022 at 7:08 PM Euler Taveira wrote: > > > > On Tue, Jun 7, 2022, at 1:10 AM, Peter Smith wrote: > > > > The logical replication tablesync ignores the publication 'publish' > > operations during the initial data copy. > > > >

RE: Replica Identity check of partition table on subscriber

2022-06-15 Thread shiy.f...@fujitsu.com
On Wed, Jun 15, 2022 8:30 PM Amit Kapila wrote: > > I have pushed the first bug-fix patch today. > Thanks. Attached the remaining patches which are rebased. Regards, Shi yu v9-0002-fix-memory-leak-about-attrmap.patch Description: v9-0002-fix-memory-leak-about-attrmap.patch

RE: Replica Identity check of partition table on subscriber

2022-06-16 Thread shiy.f...@fujitsu.com
On Thu, Jun 16, 2022 2:13 PM Amit Langote wrote: > > Hi, > > On Thu, Jun 16, 2022 at 2:07 PM shiy.f...@fujitsu.com > wrote: > > On Wed, Jun 15, 2022 8:30 PM Amit Kapila > wrote: > > > I have pushed the first bug-fix patch today. > > > > Att

RE: Replica Identity check of partition table on subscriber

2022-06-16 Thread shiy.f...@fujitsu.com
On Fri Jun 17, 2022 11:06 AM shiy.f...@fujitsu.com wrote: > > Attached the new version of patch set. I also moved the partitioned table > check > in logicalrep_rel_mark_updatable() to check_relation_updatable() as > discussed > [2]. > Attached back-branch patches of the

RE: Replica Identity check of partition table on subscriber

2022-06-14 Thread shiy.f...@fujitsu.com
On Tue, Jun 14, 2022 2:18 PM Amit Langote wrote: > > On Mon, Jun 13, 2022 at 9:26 PM Amit Kapila > wrote: > > On Mon, Jun 13, 2022 at 1:03 PM houzj.f...@fujitsu.com > > wrote: > > > On Monday, June 13, 2022 1:53 PM Amit Kapila > wrote: > > > I have separated out the bug-fix for the

RE: Replica Identity check of partition table on subscriber

2022-06-14 Thread shiy.f...@fujitsu.com
On Tue, Jun 14, 2022 8:57 PM Amit Kapila wrote: > > > v4-0002 looks good btw, except the bitpick about test comment similar > > to my earlier comment regarding v5-0001: > > > > +# Change the column order of table on publisher > > > > I think it might be better to say something specific to

RE: tablesync copy ignores publication actions

2022-06-15 Thread shiy.f...@fujitsu.com
On Tue, Jun 14, 2022 3:36 PM Peter Smith wrote: > > PSA v2 of the patch, based on all feedback received. > > ~~~ > > Main differences from v1: > > * Rewording and more explanatory text. > > * The examples were moved to the "Subscription" [1] page and also > extended to show some normal

RE: Replica Identity check of partition table on subscriber

2022-06-20 Thread shiy.f...@fujitsu.com
On Mon, Jun 20, 2022 1:33 PM Amit Kapila wrote: > > On Fri, Jun 17, 2022 at 11:22 AM shiy.f...@fujitsu.com > wrote: > > > > On Fri Jun 17, 2022 11:06 AM shiy.f...@fujitsu.com > wrote: > > > > > > Attached the new version of patch set. I al

RE: Handle infinite recursion in logical replication setup

2022-06-19 Thread shiy.f...@fujitsu.com
On Thu, Jun 16, 2022 6:18 PM vignesh C wrote: > > Thanks for the comments, the attached v21 patch has the changes for the > same. > Thanks for updating the patch. Here are some comments. 0002 patch == 1. + publisher to only send changes that originated locally. Setting +

RE: Replica Identity check of partition table on subscriber

2022-06-21 Thread shiy.f...@fujitsu.com
On Tuesday, June 21, 2022 4:49 PM Amit Kapila wrote: > > On Tue, Jun 21, 2022 at 12:50 PM Amit Langote > wrote: > > > > On Tue, Jun 21, 2022 at 3:35 PM houzj.f...@fujitsu.com > > wrote: > > > > Attached a patch containing the above to consider as an alternative. > > > > Thanks, the patch

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-25 Thread shiy.f...@fujitsu.com
Hi, I did some performance test for the master branch patch (based on v6 patch) to see if the bsearch() added by this patch will cause any overhead. I tested them three times and took the average. The results are as follows, and attach the bar chart. case 1 - No catalog modifying

RE: Handle infinite recursion in logical replication setup

2022-07-26 Thread shiy.f...@fujitsu.com
On Sun, Jul 24, 2022 1:28 AM vignesh C wrote: > > Added a note for the same and referred it to the conflicts section. > > Thanks for the comments, the attached v38 patch has the changes for the > same. > Thanks for updating the patch. A comment on the test in 0001 patch. +# Alter

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-26 Thread shiy.f...@fujitsu.com
On Tue, Jul 26, 2022 3:52 PM Masahiko Sawada wrote: > > On Tue, Jul 26, 2022 at 2:18 PM Amit Kapila > wrote: > > > > On Tue, Jul 26, 2022 at 7:00 AM Masahiko Sawada > wrote: > > > > > > On Mon, Jul 25, 2022 at 7:57 PM shiy.f...@fujitsu.com &g

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-15 Thread shiy.f...@fujitsu.com
On Mon, Jul 11, 2022 9:54 PM Masahiko Sawada wrote: > > I've attached an updated patch, please review it. > Thanks for your patch. Here are some comments for the REL14-v1 patch. 1. + Sizesz = sizeof(TransactionId) * nxacts;; There is a redundant semicolon at the

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-13 Thread shiy.f...@fujitsu.com
On Tue, Jul 12, 2022 5:23 PM Masahiko Sawada wrote: > > On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com > wrote: > > > > It happened when executing the following code because it tried to free a > NULL > > pointer (catchange_xip). > > > >

RE: Introduce wait_for_subscription_sync for TAP tests

2022-07-27 Thread shiy.f...@fujitsu.com
On Tue, Jul 26, 2022 3:42 PM Masahiko Sawada wrote: > > I've attached an updated patch as well as a patch to remove duplicated > waits in 007_ddl.pl. > Thanks for your patch. Here are some comments. 1. I think some comments need to be changed in the patch. For example: # Also wait for initial

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-17 Thread shiy.f...@fujitsu.com
On Fri, Jul 15, 2022 10:39 PM Masahiko Sawada wrote: > > This patch should have the fix for the issue that Shi yu reported. Shi > yu, could you please test it again with this patch? > Thanks for updating the patch! I have tested and confirmed that the problem I found has been fixed. Regards,

RE: Data is copied twice when specifying both child and parent table in publication

2022-04-27 Thread shiy.f...@fujitsu.com
On Sun, Apr 24, 2022 2:16 PM Wang, Wei/王 威 wrote: > > Attach the new patches.[suggestions by Amit-San] > The patch for HEAD: > 1. Add a new function to get tables info by a publications array. > The patch for REL14: > 1. Use an alias to make the statement understandable. BTW, I adjusted the >

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-08-02 Thread shiy.f...@fujitsu.com
On Mon, Aug 1, 2022 10:31 PM Amit Kapila wrote: > > On Mon, Aug 1, 2022 at 7:46 AM Masahiko Sawada > wrote: > > > > On Fri, Jul 29, 2022 at 3:45 PM Amit Kapila > wrote: > > > > > > > I've attached updated patches for all branches. Please review them. > > > > Thanks, the patches look mostly

RE: Handle infinite recursion in logical replication setup

2022-08-02 Thread shiy.f...@fujitsu.com
On Fri, Jul 29, 2022 1:22 PM vignesh C wrote: > > On Fri, Jul 29, 2022 at 8:31 AM Peter Smith > wrote: > > > > Thanks for the comments, the attached v41 patch has the changes for the > same. > Thanks for updating the patch. A comment for 0002 patch. In the example in section 31.11.4

RE: Handle infinite recursion in logical replication setup

2022-09-07 Thread shiy.f...@fujitsu.com
On Wed, Sep 7, 2022 12:23 PM vignesh C wrote: > > > Thanks for the comments, the attached v47 patch has the changes for the > same. > Thanks for updating the patch. Here is a comment. + for (i = 0; i < subrel_count; i++) + { + Oid relid =

RE: Handle infinite recursion in logical replication setup

2022-09-05 Thread shiy.f...@fujitsu.com
On Tue, Sep 6, 2022 11:14 AM vignesh C wrote: > > Thanks for the comments, the attached patch has the changes for the same. > Thanks for updating the patch. Here are some comments. 1. + if (subrel_count) + { + /* +* In case of ALTER SUBSCRIPTION ...

RE: Column Filtering in Logical Replication

2022-09-04 Thread shiy.f...@fujitsu.com
On Mon, Sep 5, 2022 8:28 AM Peter Smith wrote: > > I have rebased the remaining patch (v6-0001 is the same as v5-0002) > Thanks for updating the patch. Here are some comments. 1. + the will be successful but later + the WalSender on the publisher, or the subscriber may throw an

RE: Handle infinite recursion in logical replication setup

2022-08-31 Thread shiy.f...@fujitsu.com
On Wed, Aug 31, 2022 1:06 AM vignesh C wrote: > > The attached v43 patch has the changes for the same. > Thanks for updating the patch. Here is a comment on the 0001 patch. + if (!isnewtable) + { + pfree(nspname); +

RE: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2022-08-22 Thread shiy.f...@fujitsu.com
On Sat, Aug 20, 2022 7:02 PM Önder Kalacı wrote: > Hi, > > I'm a little late to catch up with your comments, but here are my replies: Thanks for your patch. Here are some comments. 1. In FilterOutNotSuitablePathsForReplIdentFull(), is "nonPartialIndexPathList" a good name for the list? Indexes

Fix some newly modified tab-complete changes

2022-09-27 Thread shiy.f...@fujitsu.com
Hi hackers, I saw a problem when using tab-complete for "GRANT", "TABLES IN SCHEMA" should be "ALL TABLES IN SCHEMA" in the following case. postgres=# grant all on ALL FUNCTIONS IN SCHEMA DATABASE FUNCTION PARAMETER SCHEMA

RE: Fix some newly modified tab-complete changes

2022-09-28 Thread shiy.f...@fujitsu.com
On Wed, Sep 28, 2022 1:49 PM Kyotaro Horiguchi wrote: > > At Wed, 28 Sep 2022 14:14:01 +1000, Peter Smith > wrote in > > On Tue, Sep 27, 2022 at 8:28 PM shiy.f...@fujitsu.com > > wrote: > > > > > > Hi hackers, > > > > > > I saw a problem

RE: create subscription - improved warning message

2022-10-17 Thread shiy.f...@fujitsu.com
On Mon, Oct 17, 2022 9:47 AM Peter Smith wrote: > > On Sun, Oct 16, 2022 at 12:14 AM Amit Kapila > wrote: > > > > On Fri, Oct 14, 2022 at 8:22 AM Peter Smith > wrote: > > > > > > On Thu, Oct 13, 2022 at 9:07 AM Peter Smith > wrote: > > > > > ... > > > PSA a patch for adding examples of how to

RE: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2022-10-19 Thread shiy.f...@fujitsu.com
On Wed, Oct 19, 2022 12:05 AM Önder Kalacı wrote: > > Attached v19. > Thanks for updating the patch. Here are some comments on v19. 1. In execReplication.c: + TypeCacheEntry **eq = NULL; /* only used when the index is not unique */ Maybe the comment here should be changed. Now it is

RE: Perform streaming logical transactions by background workers and parallel apply

2022-10-26 Thread shiy.f...@fujitsu.com
On Wed, Oct 26, 2022 7:19 PM Amit Kapila wrote: > > On Tue, Oct 25, 2022 at 8:38 AM Masahiko Sawada > wrote: > > > > On Fri, Oct 21, 2022 at 6:32 PM houzj.f...@fujitsu.com > > wrote: > > > > I've started to review this patch. I tested v40-0001 patch and have > > one question: > > > > IIUC even

RE: Segfault on logical replication to partitioned table with foreign children

2022-10-30 Thread shiy.f...@fujitsu.com
On Sun, Oct 30, 2022 9:39 PM Tom Lane wrote: > > What I'm wondering about is whether we can refactor this code > to avoid so many usually-useless catalog lookups. Pulling the > namespace name, in particular, is expensive and we generally > are not going to need the result. In the child-rel

RE: Perform streaming logical transactions by background workers and parallel apply

2022-10-28 Thread shiy.f...@fujitsu.com
On Tue, Oct 25, 2022 2:56 PM Wang, Wei/王 威 wrote: > > On Tues, Oct 25, 2022 at 14:28 PM Peter Smith > wrote: > > FYI - After a recent push, the v40-0001 patch no longer applies on the > > latest HEAD. > > > > [postgres@CentOS7-x64 oss_postgres_misc]$ git apply > >

RE: create subscription - improved warning message

2022-10-18 Thread shiy.f...@fujitsu.com
On Tue, Oct 18, 2022 5:44 PM Peter Smith wrote: > > On Mon, Oct 17, 2022 at 7:11 PM shiy.f...@fujitsu.com > wrote: > > > ... > > > > Thanks for your patch. Here are some comments. > > > > In Example 2, the returned slot_name should be &

RE: Perform streaming logical transactions by background workers and parallel apply

2022-09-19 Thread shiy.f...@fujitsu.com
On Mon, Sept 19, 2022 11:26 AM Wang, Wei/王 威 wrote: > > > Improved as suggested. > Thanks for updating the patch. Here are some comments on 0001 patch. 1. + case TRANS_LEADER_SERIALIZE: - oldctx = MemoryContextSwitchTo(ApplyContext); + /* +

RE: [RFC] building postgres with meson - v13

2022-10-13 Thread shiy.f...@fujitsu.com
Hi, I noticed that `pg_config --configure` didn't show the options given when building with meson. For example, meson setup build -Dcache=gcc.cache -Ddtrace=enabled -Dicu=enabled -Dcassert=true -Dprefix=/home/postgres/install_meson/ meson compile -C build meson install -C build $ pg_config

RE: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2022-10-13 Thread shiy.f...@fujitsu.com
On Wed, Aug 24, 2022 12:25 AM Önder Kalacı wrote: > Hi, > > Thanks for the review! > Thanks for your reply. > > > > > 1. > > In FilterOutNotSuitablePathsForReplIdentFull(), is > > "nonPartialIndexPathList" a > > good name for the list? Indexes on only expressions are also be filtered. > > >

RE: Fix some newly modified tab-complete changes

2022-10-10 Thread shiy.f...@fujitsu.com
On Tue, Oct 4, 2022 4:17 PM Peter Smith wrote: > > On Thu, Sep 29, 2022 at 12:50 PM shiy.f...@fujitsu.com > wrote: > > > > On Wed, Sep 28, 2022 1:49 PM Kyotaro Horiguchi > wrote: > > > > > > At Wed, 28 Sep 2022 14:14:01 +1000, Peter Smith > > &

RE: Fix some newly modified tab-complete changes

2022-10-10 Thread shiy.f...@fujitsu.com
On Mon, Oct 10, 2022 2:12 PM shiy.f...@fujitsu.com wrote: > > On Tue, Oct 4, 2022 4:17 PM Peter Smith wrote: > > > > But, while testing I noticed another different quirk > > > > It seems that neither the GRANT nor the REVOKE auto-complete > > recog

RE: [RFC] building postgres with meson - v13

2022-10-13 Thread shiy.f...@fujitsu.com
On Fri, Oct 14, 2022 12:40 AM Andres Freund wrote: > > Hi, > > On 2022-10-13 09:24:51 +, shiy.f...@fujitsu.com wrote: > > I noticed that `pg_config --configure` didn't show the options given when > > building with meson. > > Yes, that was noted somewhere o

RE: Introduce wait_for_subscription_sync for TAP tests

2022-08-04 Thread shiy.f...@fujitsu.com
On Thu, Aug 4, 2022 5:49 PM shiy.f...@fujitsu.com wrote: > > On Thu, Aug 4, 2022 2:28 PM Masahiko Sawada > wrote: > > > > On Thu, Aug 4, 2022 at 10:37 AM Amit Kapila > > wrote: > > > > > > On Wed, Aug 3, 2022 at 10:21 AM Amit Kapila > > &g

RE: Perform streaming logical transactions by background workers and parallel apply

2022-08-05 Thread shiy.f...@fujitsu.com
On Thu, Aug 4, 2022 2:36 PM Wang, Wei/王 威 wrote: > > I also did some other improvements based on the suggestions posted in this > thread. Attach the new patches. > Thanks for updating the patch. Here are some comments on v20-0001 patch. 1. +typedef struct ApplyBgworkerShared +{ +

RE: Introduce wait_for_subscription_sync for TAP tests

2022-08-04 Thread shiy.f...@fujitsu.com
On Thu, Aug 4, 2022 2:28 PM Masahiko Sawada wrote: > > On Thu, Aug 4, 2022 at 10:37 AM Amit Kapila > wrote: > > > > On Wed, Aug 3, 2022 at 10:21 AM Amit Kapila > wrote: > > > > > > Pushed this one and now I'll look at your other patch. > > > > > > > I have pushed the second patch as well after

RE: Perform streaming logical transactions by background workers and parallel apply

2022-08-17 Thread shiy.f...@fujitsu.com
On Wed, Aug 17, 2022 2:28 PM Wang, Wei/王 威 wrote: > > On Tues, August 16, 2022 15:33 PM I wrote: > > Attach the new patches. > > I found that cfbot has a failure. > After investigation, I think it is because the worker's exit state is not set > correctly. So I made some slight modifications. >

RE: Handle infinite recursion in logical replication setup

2022-07-31 Thread shiy.f...@fujitsu.com
On Fri, Jul 29, 2022 1:22 PM vignesh C wrote: > > > Thanks for the comments, the attached v41 patch has the changes for the > same. > Thanks for updating the patch. I wonder in the case that the publisher uses PG15 (or before), subscriber uses PG16, should we have this check (check if

RE: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-08-03 Thread shiy.f...@fujitsu.com
On Wed, Aug 3, 2022 12:06 PM Masahiko Sawada wrote: > > I've attached updated patches that incorporated the above comments as > well as the comments from Shi yu. Please review them. > Thanks for updating the patch. I noticed that in SnapBuildXidSetCatalogChanges(), "i" is initialized in the

RE: Force streaming every change in logical decoding

2022-12-22 Thread shiy.f...@fujitsu.com
On Thu, Dec 22, 2022 5:24 PM Amit Kapila wrote: > > On Thu, Dec 22, 2022 at 1:15 PM Kyotaro Horiguchi > wrote: > > > > At Thu, 22 Dec 2022 12:35:46 +0530, Amit Kapila > wrote in > > > I have addressed these comments in the attached. Additionally, I have > > > modified the docs and commit

RE: Force streaming every change in logical decoding

2022-12-23 Thread shiy.f...@fujitsu.com
On Fri, Dec 23, 2022 1:50 PM Amit Kapila > > On Thu, Dec 22, 2022 at 6:18 PM shiy.f...@fujitsu.com > wrote: > > > > > > Besides, I tried to reduce data size in streaming subscription tap tests by > > this > > new GUC (see 0002 patch). But I didn't cov

RE: Force streaming every change in logical decoding

2023-01-05 Thread shiy.f...@fujitsu.com
On Thu, Dec 22, 2022 3:17 PM Masahiko Sawada wrote: > > I think > > instead of adding new tests with this patch, it may be better to > > change some of the existing tests related to streaming to use this > > parameter as that will clearly show one of the purposes of this patch. > > +1. I think

RE: Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread shiy.f...@fujitsu.com
On Mon, Jan 9, 2023 11:06 PM Tom Lane wrote: > > Amit Kapila writes: > > On Mon, Jan 9, 2023 at 5:29 PM shiy.f...@fujitsu.com > > wrote: > >> I think one way to fix it is to modify pg_publication_tables query to > >> exclude > >> generated colu

RE: Allow logical replication to copy tables in binary format

2023-01-11 Thread shiy.f...@fujitsu.com
On Mon, Nov 14, 2022 8:08 PM Melih Mutlu wrote: > > Attached patch with updated version of this patch. Thanks for your patch. I tried to do a performance test for this patch, the result looks good to me. (The steps are similar to what Melih shared [1].) The time to synchronize about 1GB data

Fix pg_publication_tables to exclude generated columns

2023-01-09 Thread shiy.f...@fujitsu.com
Hi hackers, I noticed that there is a problem about system view pg_publication_tables when looking into [1]. The column "attnames" contains generated columns when no column list is specified, but generated columns shouldn't be included because they are not replicated (see

RE: Fix pg_publication_tables to exclude generated columns

2023-01-11 Thread shiy.f...@fujitsu.com
On Wed, Jan 11, 2023 2:40 PM Amit Kapila wrote: > > On Wed, Jan 11, 2023 at 10:07 AM Tom Lane wrote: > > > > Amit Kapila writes: > > >> On Mon, Jan 9, 2023 11:06 PM Tom Lane wrote: > > >>> We could just not fix it in the back branches. I'd argue that this is > > >>> as much a definition

Ignore dropped columns when checking the column list in logical replication

2023-01-03 Thread shiy.f...@fujitsu.com
Hi hackers, I saw a problem related to column list. There's a restriction that different column lists for same table can't be used in the publications of single subscription. But we will get unexpected errors in some cases because the dropped columns are not ignored. For example: -- publisher

RE: Avoid streaming the transaction which are skipped (in corner cases)

2022-12-06 Thread shiy.f...@fujitsu.com
On Wed, Dec 7, 2022 12:01 PM Dilip Kumar wrote: > > On Wed, Dec 7, 2022 at 9:28 AM Amit Kapila > wrote: > > > > On Tue, Dec 6, 2022 at 11:55 AM shiy.f...@fujitsu.com > > wrote: > > > > > > On Mon, Dec 5, 2022 6:57 PM Amit Kapila > wrote: >

RE: Force streaming every change in logical decoding

2022-12-21 Thread shiy.f...@fujitsu.com
On Wed, Dec 21, 2022 4:05 PM Peter Smith wrote: > > Here are some review comments for patch v2. > > Since the GUC is still under design maybe these comments can be > ignored for now, but I guess similar comments will apply in future > anyhow (just with some name changes). > Thanks for your

RE: Force streaming every change in logical decoding

2022-12-21 Thread shiy.f...@fujitsu.com
On Wed, Dec 21, 2022 4:54 PM Amit Kapila wrote: > > On Wed, Dec 21, 2022 at 1:55 PM Peter Smith > wrote: > > > > On Wed, Dec 21, 2022 at 6:22 PM Masahiko Sawada > wrote: > > > > > > On Tue, Dec 20, 2022 at 7:49 PM Amit Kapila > wrote: > > > > > > > > On Tue, Dec 20, 2022 at 2:46 PM Hayato

Force streaming every change in logical decoding

2022-12-05 Thread shiy.f...@fujitsu.com
Hi hackers, In logical decoding, when logical_decoding_work_mem is exceeded, the changes are sent to output plugin in streaming mode. But there is a restriction that the minimum value of logical_decoding_work_mem is 64kB. I tried to add a GUC to allow sending every change to output plugin without

RE: Avoid streaming the transaction which are skipped (in corner cases)

2022-12-05 Thread shiy.f...@fujitsu.com
On Mon, Dec 5, 2022 6:57 PM Amit Kapila wrote: > > On Mon, Dec 5, 2022 at 3:41 PM Dilip Kumar wrote: > > > > I think we need something like this[1] so that we can better control > > the streaming. > > > > +1. The additional advantage would be that we can generate parallel > apply and new

RE: Force streaming every change in logical decoding

2022-12-14 Thread shiy.f...@fujitsu.com
On Sat, Dec 10, 2022 2:03 PM Dilip Kumar wrote: > > On Tue, Dec 6, 2022 at 11:53 AM shiy.f...@fujitsu.com > wrote: > > > > Hi hackers, > > > > In logical decoding, when logical_decoding_work_mem is exceeded, the > changes are > &g

RE: Avoid streaming the transaction which are skipped (in corner cases)

2022-11-28 Thread shiy.f...@fujitsu.com
On Sun, Nov 27, 2022 1:33 PM Dilip Kumar wrote: > > On Sat, Nov 26, 2022 at 12:15 PM Amit Kapila > wrote: > > > > On Fri, Nov 25, 2022 at 5:38 PM Amit Kapila > wrote: > > > > > > On Fri, Nov 25, 2022 at 1:35 PM Dilip Kumar > wrote: > > > > > > > > During DecodeCommit() for skipping a

  1   2   >