Re: bogus: logical replication rows/cols combinations

2022-06-15 Thread Amit Kapila
On Mon, Jun 13, 2022 at 8:54 AM Amit Kapila wrote: > > I would like to close the Open item listed corresponding to this > thread [1] as the fix for the reported issue is committed > (fd0b9dcebd). Do let me know if you or others think otherwise? > Seeing no objections, I have closed this item. --

Re: bogus: logical replication rows/cols combinations

2022-06-12 Thread Amit Kapila
On Wed, Jun 8, 2022 at 11:05 AM Peter Smith wrote: > > On Wed, Jun 8, 2022 at 1:25 PM Justin Pryzby wrote: > > > > On Mon, Jun 06, 2022 at 03:42:31PM +1000, Peter Smith wrote: > > > I noticed the patch "0001-language-fixes-on-HEAD-from-Justin.patch" says: > > > > > > @@ -11673,7 +11673,7 @@ > > >

Re: bogus: logical replication rows/cols combinations

2022-06-07 Thread Peter Smith
On Wed, Jun 8, 2022 at 1:25 PM Justin Pryzby wrote: > > On Mon, Jun 06, 2022 at 03:42:31PM +1000, Peter Smith wrote: > > I noticed the patch "0001-language-fixes-on-HEAD-from-Justin.patch" says: > > > > @@ -11673,7 +11673,7 @@ > >prosrc => 'pg_show_replication_origin_status' }, > > > > # publ

Re: bogus: logical replication rows/cols combinations

2022-06-07 Thread Justin Pryzby
On Mon, Jun 06, 2022 at 03:42:31PM +1000, Peter Smith wrote: > I noticed the patch "0001-language-fixes-on-HEAD-from-Justin.patch" says: > > @@ -11673,7 +11673,7 @@ >prosrc => 'pg_show_replication_origin_status' }, > > # publications > -{ oid => '6119', descr => 'get information of tables in

Re: bogus: logical replication rows/cols combinations

2022-06-05 Thread Peter Smith
On Thu, Jun 2, 2022 at 9:58 PM Amit Kapila wrote: > > On Fri, May 27, 2022 at 1:04 PM houzj.f...@fujitsu.com > wrote: > > > > On Friday, May 27, 2022 1:54 PM Justin Pryzby wrote: > > > > > > On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote: > > > > On Tue, May 24, 2022 at 11:03 AM hou

Re: bogus: logical replication rows/cols combinations

2022-06-02 Thread Amit Kapila
On Fri, May 27, 2022 at 1:04 PM houzj.f...@fujitsu.com wrote: > > On Friday, May 27, 2022 1:54 PM Justin Pryzby wrote: > > > > On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote: > > > On Tue, May 24, 2022 at 11:03 AM houzj.f...@fujitsu.com > > wrote: > > > > > > > > On Friday, May 20,

RE: bogus: logical replication rows/cols combinations

2022-05-27 Thread houzj.f...@fujitsu.com
On Friday, May 27, 2022 1:54 PM Justin Pryzby wrote: > > On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote: > > On Tue, May 24, 2022 at 11:03 AM houzj.f...@fujitsu.com > wrote: > > > > > > On Friday, May 20, 2022 11:06 AM Amit Kapila > wrote: > > > > > > Thanks for pointing it out. He

Re: bogus: logical replication rows/cols combinations

2022-05-26 Thread Justin Pryzby
On Fri, May 27, 2022 at 11:17:00AM +0530, Amit Kapila wrote: > On Tue, May 24, 2022 at 11:03 AM houzj.f...@fujitsu.com > wrote: > > > > On Friday, May 20, 2022 11:06 AM Amit Kapila > > wrote: > > > > Thanks for pointing it out. Here is the new version patch which add this > > version check. >

Re: bogus: logical replication rows/cols combinations

2022-05-26 Thread Amit Kapila
On Tue, May 24, 2022 at 11:03 AM houzj.f...@fujitsu.com wrote: > > On Friday, May 20, 2022 11:06 AM Amit Kapila wrote: > > Thanks for pointing it out. Here is the new version patch which add this > version check. > I have added/edited a few comments and ran pgindent. The attached looks good to

Re: bogus: logical replication rows/cols combinations

2022-05-25 Thread Amit Kapila
On Tue, May 24, 2022 at 3:19 PM Amit Kapila wrote: > > On Fri, May 20, 2022 at 8:36 AM Amit Kapila wrote: > > > > On Thu, May 19, 2022 at 7:54 PM Tom Lane wrote: > > > > > > Justin Pryzby writes: > > > > On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote: > > > >> I have committed the

Re: bogus: logical replication rows/cols combinations

2022-05-24 Thread Amit Kapila
On Fri, May 20, 2022 at 8:36 AM Amit Kapila wrote: > > On Thu, May 19, 2022 at 7:54 PM Tom Lane wrote: > > > > Justin Pryzby writes: > > > On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote: > > >> I have committed the first patch after fixing this part. It seems Tom > > >> is not very

RE: bogus: logical replication rows/cols combinations

2022-05-23 Thread houzj.f...@fujitsu.com
On Friday, May 20, 2022 11:06 AM Amit Kapila wrote: > > On Thu, May 19, 2022 at 7:54 PM Tom Lane wrote: > > > Even more to the point, how can we > > have a subscriber do that by relying on view columns that don't exist > > in older versions? > > > > We need a version check like (if > (walrcv_

Re: bogus: logical replication rows/cols combinations

2022-05-19 Thread Amit Kapila
On Thu, May 19, 2022 at 7:54 PM Tom Lane wrote: > > Justin Pryzby writes: > > On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote: > >> I have committed the first patch after fixing this part. It seems Tom > >> is not very happy doing this after beta-1 [1]. The reason we get this > >> inf

Re: bogus: logical replication rows/cols combinations

2022-05-19 Thread Tom Lane
Justin Pryzby writes: > On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote: >> I have committed the first patch after fixing this part. It seems Tom >> is not very happy doing this after beta-1 [1]. The reason we get this >> information via this view (and underlying function) is that it >

Re: bogus: logical replication rows/cols combinations

2022-05-19 Thread Justin Pryzby
On Thu, May 19, 2022 at 10:33:13AM +0530, Amit Kapila wrote: > I have committed the first patch after fixing this part. It seems Tom > is not very happy doing this after beta-1 [1]. The reason we get this > information via this view (and underlying function) is that it > simplifies the queries on t

Re: bogus: logical replication rows/cols combinations

2022-05-18 Thread Amit Kapila
On Mon, May 16, 2022 at 6:50 PM Alvaro Herrera wrote: > > On 2022-May-16, Amit Kapila wrote: > > > Few comments: > > = > > 1. > > postgres=# select * from pg_publication_tables; > > pubname | schemaname | tablename | columnlist | rowfilter > > -++---+-

Re: bogus: logical replication rows/cols combinations

2022-05-17 Thread Amit Kapila
On Tue, May 17, 2022 at 2:40 PM houzj.f...@fujitsu.com wrote: > > Attach the new version patch which addressed all the above comments and > comments from Shi yu[1] and Osumi-san[2]. > Thanks, your first patch looks good to me. I'll commit that tomorrow unless there are more comments on the same.

RE: bogus: logical replication rows/cols combinations

2022-05-17 Thread houzj.f...@fujitsu.com
On Tuesday, May 17, 2022 2:53 PM Amit Kapila wrote: > > Few minor comments: > == > 1. > + > + Names of table columns included in the publication. This contains all > + the columns of table when user didn't specify column list for the > + table. > + >

Re: bogus: logical replication rows/cols combinations

2022-05-16 Thread Amit Kapila
On Mon, May 16, 2022 at 6:04 PM houzj.f...@fujitsu.com wrote: > > Attach the new version patch. > Few minor comments: == 1. + + Names of table columns included in the publication. This contains all + the columns of table when user didn't specify column list for t

RE: bogus: logical replication rows/cols combinations

2022-05-16 Thread osumi.takami...@fujitsu.com
On Monday, May 16, 2022 9:34 PM houzj.f...@fujitsu.com wrote: > Attach the new version patch. Hi, I have few minor comments. For v2-0001. (1) Unnecessary period at the end of column explanation + + + rowfilter text + + + Expression for the table's publicati

Re: bogus: logical replication rows/cols combinations

2022-05-16 Thread Amit Kapila
On Mon, May 16, 2022 at 6:50 PM Alvaro Herrera wrote: > > On 2022-May-16, Amit Kapila wrote: > > > > Agreed. If we can get columnlist and rowfilter from > > > pg_publication_tables, it > > > will be more convenient. And I think users that want to fetch the > > > columnlist > > > and rowfilter of

RE: bogus: logical replication rows/cols combinations

2022-05-16 Thread shiy.f...@fujitsu.com
On Mon, May 16, 2022 8:34 PM houzj.f...@fujitsu.com wrote: > > Attach the new version patch. > Thanks for your patch. Here are some comments: 1. (0001 patch) /* * Returns Oids of tables in a publication. */ Datum pg_get_publication_tables(PG_FUNCTION_ARGS) Should we modify the comment of p

Re: bogus: logical replication rows/cols combinations

2022-05-16 Thread Alvaro Herrera
On 2022-May-16, Amit Kapila wrote: > > Agreed. If we can get columnlist and rowfilter from pg_publication_tables, > > it > > will be more convenient. And I think users that want to fetch the columnlist > > and rowfilter of table can also benefit from it. > > After the change for this, we will gi

RE: bogus: logical replication rows/cols combinations

2022-05-16 Thread houzj.f...@fujitsu.com
On Monday, May 16, 2022 2:10 PM Amit Kapila > > On Fri, May 13, 2022 at 11:32 AM houzj.f...@fujitsu.com > wrote: > > > > On Thursday, May 12, 2022 2:45 PM Amit Kapila > wrote: > > > > > > On Wed, May 11, 2022 at 12:55 PM houzj.f...@fujitsu.com > > > wrote: > > > > > > Few comments: > > > =

Re: bogus: logical replication rows/cols combinations

2022-05-15 Thread Amit Kapila
On Fri, May 13, 2022 at 11:32 AM houzj.f...@fujitsu.com wrote: > > On Thursday, May 12, 2022 2:45 PM Amit Kapila wrote: > > > > On Wed, May 11, 2022 at 12:55 PM houzj.f...@fujitsu.com > > wrote: > > > > Few comments: > > === > > 1. > > initStringInfo(&cmd); > > - appendStringInfoStri

RE: bogus: logical replication rows/cols combinations

2022-05-12 Thread houzj.f...@fujitsu.com
On Thursday, May 12, 2022 2:45 PM Amit Kapila wrote: > > On Wed, May 11, 2022 at 12:55 PM houzj.f...@fujitsu.com > wrote: > > > > On Wednesday, May 11, 2022 11:33 AM Amit Kapila > wrote: > > > > > > Fair enough, then we should go with a simpler approach to detect it > > > in pgoutput.c (get_rel

Re: bogus: logical replication rows/cols combinations

2022-05-12 Thread Amit Kapila
On Thu, May 12, 2022 at 12:15 PM Amit Kapila wrote: > > On Wed, May 11, 2022 at 12:55 PM houzj.f...@fujitsu.com > wrote: > > > > On Wednesday, May 11, 2022 11:33 AM Amit Kapila > > wrote: > > > > > > Fair enough, then we should go with a simpler approach to detect it in > > > pgoutput.c (get_re

Re: bogus: logical replication rows/cols combinations

2022-05-11 Thread Amit Kapila
On Wed, May 11, 2022 at 12:55 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, May 11, 2022 11:33 AM Amit Kapila > wrote: > > > > Fair enough, then we should go with a simpler approach to detect it in > > pgoutput.c (get_rel_sync_entry). > > OK, here is the patch that try to check column list

RE: bogus: logical replication rows/cols combinations

2022-05-11 Thread houzj.f...@fujitsu.com
On Wednesday, May 11, 2022 11:33 AM Amit Kapila wrote: > On Wed, May 11, 2022 at 12:35 AM Tomas Vondra > wrote: > > > > On 5/9/22 05:45, Amit Kapila wrote: > > > On Sun, May 8, 2022 at 11:41 PM Tomas Vondra > > > wrote: > > >> > > >> On 5/7/22 07:36, Amit Kapila wrote: > > >>> On Mon, May 2, 202

Re: bogus: logical replication rows/cols combinations

2022-05-10 Thread Amit Kapila
On Wed, May 11, 2022 at 12:35 AM Tomas Vondra wrote: > > On 5/9/22 05:45, Amit Kapila wrote: > > On Sun, May 8, 2022 at 11:41 PM Tomas Vondra > > wrote: > >> > >> On 5/7/22 07:36, Amit Kapila wrote: > >>> On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera > >>> wrote: > > On 2022-May-02, A

Re: bogus: logical replication rows/cols combinations

2022-05-10 Thread Tomas Vondra
On 5/9/22 05:45, Amit Kapila wrote: > On Sun, May 8, 2022 at 11:41 PM Tomas Vondra > wrote: >> >> On 5/7/22 07:36, Amit Kapila wrote: >>> On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera >>> wrote: On 2022-May-02, Amit Kapila wrote: > I think it is possible to expose a li

Re: bogus: logical replication rows/cols combinations

2022-05-08 Thread Amit Kapila
On Sun, May 8, 2022 at 11:41 PM Tomas Vondra wrote: > > On 5/7/22 07:36, Amit Kapila wrote: > > On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera > > wrote: > >> > >> On 2022-May-02, Amit Kapila wrote: > >> > >> > >>> I think it is possible to expose a list of publications for each > >>> walsender a

Re: bogus: logical replication rows/cols combinations

2022-05-08 Thread Tomas Vondra
On 5/7/22 07:36, Amit Kapila wrote: > On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera wrote: >> >> On 2022-May-02, Amit Kapila wrote: >> >> >>> I think it is possible to expose a list of publications for each >>> walsender as it is stored in each walsenders >>> LogicalDecodingContext->output_plugin_

Re: bogus: logical replication rows/cols combinations

2022-05-06 Thread Amit Kapila
On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera wrote: > > On 2022-May-02, Amit Kapila wrote: > > > > I think it is possible to expose a list of publications for each > > walsender as it is stored in each walsenders > > LogicalDecodingContext->output_plugin_private. AFAIK, each walsender > > can hav

Re: bogus: logical replication rows/cols combinations

2022-05-06 Thread Tomas Vondra
On 5/6/22 15:40, Amit Kapila wrote: > On Fri, May 6, 2022 at 5:56 PM Tomas Vondra > wrote: >> I could think of below two options: 1. Forbid any case where column list is different for the same table when combining publications. 2. Forbid if the column list and row filte

Re: bogus: logical replication rows/cols combinations

2022-05-06 Thread Amit Kapila
On Fri, May 6, 2022 at 5:56 PM Tomas Vondra wrote: > > >> > >> I could think of below two options: > >> 1. Forbid any case where column list is different for the same table > >> when combining publications. > >> 2. Forbid if the column list and row filters for a table are different > >> in the set

Re: bogus: logical replication rows/cols combinations

2022-05-06 Thread Tomas Vondra
On 5/6/22 05:23, houzj.f...@fujitsu.com wrote: > On Tuesday, May 3, 2022 11:31 AM Amit Kapila wrote: >> >> On Tue, May 3, 2022 at 12:10 AM Tomas Vondra >> wrote: >>> >>> On 5/2/22 19:51, Alvaro Herrera wrote: > Why would we need to know publications replicated by other >> walsenders? >

RE: bogus: logical replication rows/cols combinations

2022-05-05 Thread houzj.f...@fujitsu.com
On Tuesday, May 3, 2022 11:31 AM Amit Kapila wrote: > > On Tue, May 3, 2022 at 12:10 AM Tomas Vondra > wrote: > > > > On 5/2/22 19:51, Alvaro Herrera wrote: > > >> Why would we need to know publications replicated by other > walsenders? > > >> And what if the subscriber is not connected at the m

Re: bogus: logical replication rows/cols combinations

2022-05-04 Thread Peter Eisentraut
On 03.05.22 21:40, Tomas Vondra wrote: So what's wrong with merging the column lists as implemented in the v2 patch, posted a couple days ago? Merging the column lists is ok if all other publication attributes match. Otherwise, I think not. I don't think triggers are a suitable alternati

Re: bogus: logical replication rows/cols combinations

2022-05-03 Thread Tomas Vondra
On 5/2/22 22:34, Peter Eisentraut wrote: > On 01.05.22 23:42, Tomas Vondra wrote: >> Imagine have a table with customers from different regions, and you want >> to replicate the data somewhere else, but for some reason you can only >> replicate details for one particular region, and subset of co

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 6:11 PM Alvaro Herrera wrote: > > On 2022-May-02, Amit Kapila wrote: > > > I think it is possible to expose a list of publications for each > > walsender as it is stored in each walsenders > > LogicalDecodingContext->output_plugin_private. AFAIK, each walsender > > can have

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Tue, May 3, 2022 at 12:10 AM Tomas Vondra wrote: > > On 5/2/22 19:51, Alvaro Herrera wrote: > >> Why would we need to know publications replicated by other walsenders? > >> And what if the subscriber is not connected at the moment? In that case > >> there'll be no walsender. > > > > Sure, if th

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Peter Eisentraut
On 01.05.22 23:42, Tomas Vondra wrote: Imagine have a table with customers from different regions, and you want to replicate the data somewhere else, but for some reason you can only replicate details for one particular region, and subset of columns for everyone else. So you'd do something like t

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 19:51, Alvaro Herrera wrote: > On 2022-May-02, Tomas Vondra wrote: > >> pgoutput.c is relies on relcache callbacks to get notified of changes. >> See the stuff that touches replicate_valid and publications_valid. So >> the walsender should notice the changes immediately. > > Hmm, I s

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 13:23, Amit Kapila wrote: > On Mon, May 2, 2022 at 3:05 PM Tomas Vondra > wrote: >> >> On 5/2/22 07:31, Amit Kapila wrote: >>> On Mon, May 2, 2022 at 3:27 AM Tomas Vondra >>> wrote: >> The second option has the annoying consequence that it makes this useless for the "d

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Tomas Vondra wrote: > pgoutput.c is relies on relcache callbacks to get notified of changes. > See the stuff that touches replicate_valid and publications_valid. So > the walsender should notice the changes immediately. Hmm, I suppose that makes any changes easy enough to detect.

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 13:44, Alvaro Herrera wrote: > On 2022-May-02, Amit Kapila wrote: > >> We don't do that currently but we can as mentioned in my previous >> email [1]. Let me write the relevant part again. We need to expose all >> publications for a walsender, and then we can find the exact set of >>

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-Apr-28, Tomas Vondra wrote: > SELECT > (CASE WHEN (a < 0) OR (a > 0) THEN a ELSE NULL END) AS a, > (CASE WHEN (a > 0) THEN b ELSE NULL END) AS b, > (CASE WHEN (a < 0) THEN c ELSE NULL END) AS c > FROM uno WHERE (a < 0) OR (a > 0) BTW, looking at the new COPY commands, the

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Amit Kapila wrote: > We don't do that currently but we can as mentioned in my previous > email [1]. Let me write the relevant part again. We need to expose all > publications for a walsender, and then we can find the exact set of > publications where the current publication is used

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 3:05 PM Tomas Vondra wrote: > > On 5/2/22 07:31, Amit Kapila wrote: > > On Mon, May 2, 2022 at 3:27 AM Tomas Vondra > > wrote: > >> > > >> The second option has the annoying consequence that it makes this > >> useless for the "data redaction" use case I described in [2], be

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Amit Kapila
On Mon, May 2, 2022 at 3:53 PM Tomas Vondra wrote: > > On 5/2/22 12:17, Alvaro Herrera wrote: > > On 2022-May-02, Tomas Vondra wrote: > >> On 5/2/22 07:31, Amit Kapila wrote: > > > >>> Yeah, or don't allow to define such publications in the first place so > >>> that different subscriptions can't c

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Tomas Vondra wrote: > On 5/2/22 12:17, Alvaro Herrera wrote: > > The latter ultimately means that we aren't sure that a combined > > subscription is safe. And in turn this means that a pg_dump of such a > > database cannot be restored (because the CREATE SUBSCRIPTION will be > >

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 12:17, Alvaro Herrera wrote: > On 2022-May-02, Tomas Vondra wrote: >> On 5/2/22 07:31, Amit Kapila wrote: > >>> Yeah, or don't allow to define such publications in the first place so >>> that different subscriptions can't combine them but I guess that might >>> forbid some useful case

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Alvaro Herrera
On 2022-May-02, Tomas Vondra wrote: > On 5/2/22 07:31, Amit Kapila wrote: > > Yeah, or don't allow to define such publications in the first place so > > that different subscriptions can't combine them but I guess that might > > forbid some useful cases as well where publication may not get > > com

Re: bogus: logical replication rows/cols combinations

2022-05-02 Thread Tomas Vondra
On 5/2/22 07:31, Amit Kapila wrote: > On Mon, May 2, 2022 at 3:27 AM Tomas Vondra > wrote: >> >> On 4/30/22 12:11, Amit Kapila wrote: >>> On Sat, Apr 30, 2022 at 3:01 PM Alvaro Herrera >>> wrote: My proposal is that if users want to define multiple publications, and their defin

Re: bogus: logical replication rows/cols combinations

2022-05-01 Thread Amit Kapila
On Mon, May 2, 2022 at 11:01 AM Amit Kapila wrote: > > On Mon, May 2, 2022 at 3:27 AM Tomas Vondra > wrote: > > > > On 4/30/22 12:11, Amit Kapila wrote: > > > On Sat, Apr 30, 2022 at 3:01 PM Alvaro Herrera > > > wrote: > > >> > > >> My proposal is that if users want to define multiple publicati

Re: bogus: logical replication rows/cols combinations

2022-05-01 Thread Amit Kapila
On Mon, May 2, 2022 at 3:27 AM Tomas Vondra wrote: > > On 4/30/22 12:11, Amit Kapila wrote: > > On Sat, Apr 30, 2022 at 3:01 PM Alvaro Herrera > > wrote: > >> > >> My proposal is that if users want to define multiple publications, and > >> their definitions conflict in a way that would behave ri

Re: bogus: logical replication rows/cols combinations

2022-05-01 Thread Tomas Vondra
On 4/30/22 12:11, Amit Kapila wrote: > On Sat, Apr 30, 2022 at 3:01 PM Alvaro Herrera > wrote: >> >> On 2022-Apr-30, Amit Kapila wrote: >> >>> On Sat, Apr 30, 2022 at 2:02 AM Tomas Vondra >>> wrote: >> That seems to deal with a circular replication, i.e. two logical replication links -

Re: bogus: logical replication rows/cols combinations

2022-05-01 Thread Tomas Vondra
On 4/29/22 07:05, Amit Kapila wrote: > On Thu, Apr 28, 2022 at 5:56 PM Peter Eisentraut > wrote: >> >> On 27.04.22 12:33, Amit Kapila wrote: >>> Currently, when the subscription has multiple publications, we combine >>> the objects, and actions of those publications. It happens for >>> 'publish

Re: bogus: logical replication rows/cols combinations

2022-05-01 Thread Tomas Vondra
On 4/30/22 11:28, Alvaro Herrera wrote: > On 2022-Apr-28, Tomas Vondra wrote: > >> Attached is a patch doing the same thing in tablesync. The overall idea >> is to generate copy statement with CASE expressions, applying filters to >> individual columns. For Alvaro's example, this generates some

Re: bogus: logical replication rows/cols combinations

2022-04-30 Thread Alvaro Herrera
On 2022-Apr-30, Amit Kapila wrote: > I agree with throwing errors for obvious/known bogus cases but do we > want to throw errors or restrict the combining of column lists when > row filters are present in all cases? See some examples [1 ] where it > may be valid to combine them. I agree we should

Re: bogus: logical replication rows/cols combinations

2022-04-30 Thread Amit Kapila
On Sat, Apr 30, 2022 at 3:01 PM Alvaro Herrera wrote: > > On 2022-Apr-30, Amit Kapila wrote: > > > On Sat, Apr 30, 2022 at 2:02 AM Tomas Vondra > > wrote: > > > > That seems to deal with a circular replication, i.e. two logical > > > replication links - a bit like a multi-master. Not sure how is

Re: bogus: logical replication rows/cols combinations

2022-04-30 Thread Alvaro Herrera
On 2022-Apr-30, Amit Kapila wrote: > On Sat, Apr 30, 2022 at 2:02 AM Tomas Vondra > wrote: > > That seems to deal with a circular replication, i.e. two logical > > replication links - a bit like a multi-master. Not sure how is that > > related to the issue we're discussing here? > > It is not d

Re: bogus: logical replication rows/cols combinations

2022-04-30 Thread Alvaro Herrera
On 2022-Apr-28, Tomas Vondra wrote: > Attached is a patch doing the same thing in tablesync. The overall idea > is to generate copy statement with CASE expressions, applying filters to > individual columns. For Alvaro's example, this generates something like > > SELECT > (CASE WHEN (a < 0)

Re: bogus: logical replication rows/cols combinations

2022-04-29 Thread Amit Kapila
On Sat, Apr 30, 2022 at 2:02 AM Tomas Vondra wrote: > > On 4/29/22 06:48, Amit Kapila wrote: > > On Thu, Apr 28, 2022 at 11:00 PM Tomas Vondra > > I think such issues due to ALTER of the publication are somewhat > expected, and I think users will understand they might need to resync > the subscrip

Re: bogus: logical replication rows/cols combinations

2022-04-29 Thread Tomas Vondra
On 4/29/22 06:48, Amit Kapila wrote: > On Thu, Apr 28, 2022 at 11:00 PM Tomas Vondra > wrote: >> >> On 4/28/22 14:26, Peter Eisentraut wrote: >>> On 27.04.22 12:33, Amit Kapila wrote: >>> >>> I wonder how we handle the combination of >>> >>> pub1: publish=insert WHERE (foo) >>> pub2: publish=updat

Re: bogus: logical replication rows/cols combinations

2022-04-28 Thread Amit Kapila
On Thu, Apr 28, 2022 at 5:56 PM Peter Eisentraut wrote: > > On 27.04.22 12:33, Amit Kapila wrote: > > Currently, when the subscription has multiple publications, we combine > > the objects, and actions of those publications. It happens for > > 'publish_via_partition_root', publication actions, tab

Re: bogus: logical replication rows/cols combinations

2022-04-28 Thread Amit Kapila
On Thu, Apr 28, 2022 at 11:00 PM Tomas Vondra wrote: > > On 4/28/22 14:26, Peter Eisentraut wrote: > > On 27.04.22 12:33, Amit Kapila wrote: > > > > I wonder how we handle the combination of > > > > pub1: publish=insert WHERE (foo) > > pub2: publish=update WHERE (bar) > > > > I think it would be i

Re: bogus: logical replication rows/cols combinations

2022-04-28 Thread Tomas Vondra
On 4/28/22 14:26, Peter Eisentraut wrote: > On 27.04.22 12:33, Amit Kapila wrote: >> Currently, when the subscription has multiple publications, we combine >> the objects, and actions of those publications. It happens for >> 'publish_via_partition_root', publication actions, tables, column >> lists

Re: bogus: logical replication rows/cols combinations

2022-04-28 Thread Tomas Vondra
On 4/28/22 05:17, Amit Kapila wrote: > On Thu, Apr 28, 2022 at 3:26 AM Tomas Vondra > wrote: >> >> so I've been looking at tweaking the code so that the behavior matches >> Alvaro's expectations. It passes check-world but I'm not claiming it's >> nowhere near commitable - the purpose is mostly t

Re: bogus: logical replication rows/cols combinations

2022-04-28 Thread Peter Eisentraut
On 27.04.22 12:33, Amit Kapila wrote: Currently, when the subscription has multiple publications, we combine the objects, and actions of those publications. It happens for 'publish_via_partition_root', publication actions, tables, column lists, or row filters. I think the whole design works on th

Re: bogus: logical replication rows/cols combinations

2022-04-28 Thread Peter Eisentraut
On 27.04.22 11:53, Alvaro Herrera wrote: Now, another possibility is to say "naah, this is too hard", or even "naah, there's no time to write all that for this release". That might be okay, but in that case let's add an implementation restriction to ensure that we don't paint ourselves in a corn

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Amit Kapila
On Thu, Apr 28, 2022 at 3:26 AM Tomas Vondra wrote: > > so I've been looking at tweaking the code so that the behavior matches > Alvaro's expectations. It passes check-world but I'm not claiming it's > nowhere near commitable - the purpose is mostly to give better idea of > how invasive the change

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Tomas Vondra
Hi, so I've been looking at tweaking the code so that the behavior matches Alvaro's expectations. It passes check-world but I'm not claiming it's nowhere near commitable - the purpose is mostly to give better idea of how invasive the change is etc. As described earlier, this abandons the idea of

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Amit Kapila
On Wed, Apr 27, 2022 at 4:27 PM Alvaro Herrera wrote: > > On 2022-Apr-27, Amit Kapila wrote: > > > On Wed, Apr 27, 2022 at 3:13 PM Alvaro Herrera > > wrote: > > > > > Changing this to behave the way you expect would be quite difficult, > > > > because at the moment we build a single OR expressio

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Alvaro Herrera
On 2022-Apr-27, Amit Kapila wrote: > On Wed, Apr 27, 2022 at 3:13 PM Alvaro Herrera > wrote: > > > Changing this to behave the way you expect would be quite difficult, > > > because at the moment we build a single OR expression from all the row > > > filters. We'd have to keep the individual ex

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Amit Kapila
On Wed, Apr 27, 2022 at 3:13 PM Alvaro Herrera wrote: > > On 2022-Apr-26, Tomas Vondra wrote: > > > I'm not quite sure which of the two behaviors is more "desirable". In a > > way, it's somewhat similar to publish_as_relid, which is also calculated > > not considering which of the row filters matc

RE: bogus: logical replication rows/cols combinations

2022-04-27 Thread houzj.f...@fujitsu.com
On Wednesday, April 27, 2022 12:56 PM From: Amit Kapila wrote: > On Tue, Apr 26, 2022 at 4:00 AM Tomas Vondra > wrote: > > > > On 4/25/22 17:48, Alvaro Herrera wrote: > > > > > The desired result on subscriber is: > > > > > > table uno; > > > a │ b │ c > > > ┼───┼─── > > > 1 │ 2 │ > > >

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Alvaro Herrera
On 2022-Apr-27, Amit Kapila wrote: > On Tue, Apr 26, 2022 at 4:00 AM Tomas Vondra > wrote: > > I can take a stab at it, but it seems strange to not apply the same > > logic to evaluation of publish_as_relid. > > Yeah, the current behavior seems to be consistent with what we already > do. Sorry

Re: bogus: logical replication rows/cols combinations

2022-04-27 Thread Alvaro Herrera
On 2022-Apr-26, Tomas Vondra wrote: > I'm not quite sure which of the two behaviors is more "desirable". In a > way, it's somewhat similar to publish_as_relid, which is also calculated > not considering which of the row filters match? I grepped doc/src/sgml for `publish_as_relid` and found no hit

Re: bogus: logical replication rows/cols combinations

2022-04-26 Thread Michael Paquier
On Wed, Apr 27, 2022 at 10:25:50AM +0530, Amit Kapila wrote: > I feel we can explain a bit more about this in docs. We already have > some explanation of how row filters are combined [1]. We can probably > add a few examples for column lists. I am not completely sure exactly what we should do here

Re: bogus: logical replication rows/cols combinations

2022-04-26 Thread Amit Kapila
On Tue, Apr 26, 2022 at 4:00 AM Tomas Vondra wrote: > > On 4/25/22 17:48, Alvaro Herrera wrote: > > > The desired result on subscriber is: > > > > table uno; > > a │ b │ c > > ┼───┼─── > > 1 │ 2 │ > > -1 │ │ 4 > > > > > > Thoughts? > > > > I'm not quite sure which of the two behaviors i

Re: bogus: logical replication rows/cols combinations

2022-04-25 Thread Tomas Vondra
On 4/25/22 17:48, Alvaro Herrera wrote: > I just noticed that publishing tables on multiple publications with > different row filters and column lists has somewhat surprising behavior. > To wit: if a column is published in any row-filtered publication, then > the values for that column are sent to