Re: parallel vacuum comments

2021-12-21 Thread Masahiko Sawada
On Tue, Dec 21, 2021 at 10:24 PM Amit Kapila wrote: > > On Tue, Dec 21, 2021 at 11:24 AM Masahiko Sawada > wrote: > > > > On Tue, Dec 21, 2021 at 2:04 PM Amit Kapila wrote: > > > > > > > Thank you for the comment. Agreed. > > > > I'v

Re: do only critical work during single-user vacuum?

2021-12-21 Thread Masahiko Sawada
enxid/relminmxid and want to manually run vacuum on them rather than relying on autovacuums. --min-xid-age option and --min-mxid-age option of vacuumdb command would be good examples. So I think this new command/facility might not necessarily need to be specific to single-user mode. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: parallel vacuum comments

2021-12-21 Thread Masahiko Sawada
On Tue, Dec 21, 2021 at 10:24 PM Amit Kapila wrote: > > On Tue, Dec 21, 2021 at 11:24 AM Masahiko Sawada > wrote: > > > > On Tue, Dec 21, 2021 at 2:04 PM Amit Kapila wrote: > > > > > > > Thank you for the comment. Agreed. > > > > I'v

Re: do only critical work during single-user vacuum?

2021-12-20 Thread Masahiko Sawada
On Tue, Dec 21, 2021 at 1:53 PM Peter Geoghegan wrote: > > On Mon, Dec 20, 2021 at 8:40 PM Masahiko Sawada wrote: > > BTW a vacuum automatically enters failsafe mode under the situation > > where the user has to run a vacuum in the single-user mode, right? > > Onl

Re: parallel vacuum comments

2021-12-20 Thread Masahiko Sawada
On Tue, Dec 21, 2021 at 2:04 PM Amit Kapila wrote: > > On Tue, Dec 21, 2021 at 10:05 AM Masahiko Sawada > wrote: > > > > On Tue, Dec 21, 2021 at 12:05 PM Amit Kapila > > wrote: > > > > > > On Mon, Dec 20, 2021 at 6:29 PM Masahiko S

Re: do only critical work during single-user vacuum?

2021-12-20 Thread Masahiko Sawada
matically enters failsafe mode under the situation where the user has to run a vacuum in the single-user mode, right? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: parallel vacuum comments

2021-12-20 Thread Masahiko Sawada
On Tue, Dec 21, 2021 at 12:05 PM Amit Kapila wrote: > > On Mon, Dec 20, 2021 at 6:29 PM Masahiko Sawada wrote: > > > > On Mon, Dec 20, 2021 at 1:08 PM Amit Kapila wrote: > > > > > > > > > > > > > 2. What is the reason for not movi

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-12-20 Thread Masahiko Sawada
On Sat, Dec 18, 2021 at 11:29 AM Peter Geoghegan wrote: > > On Thu, Dec 16, 2021 at 10:46 PM Masahiko Sawada > wrote: > > > My emphasis here has been on making non-aggressive VACUUMs *always* > > > advance relfrozenxid, outside of certain obvious edge cases. And so

Re: parallel vacuum comments

2021-12-20 Thread Masahiko Sawada
On Mon, Dec 20, 2021 at 1:08 PM Amit Kapila wrote: > > On Mon, Dec 20, 2021 at 8:33 AM Masahiko Sawada wrote: > > > > On Sat, Dec 18, 2021 at 3:38 PM Amit Kapila wrote: > > > > > > Few comments: > > > = > > > 1. > >

Re: parallel vacuum comments

2021-12-19 Thread Masahiko Sawada
On Sat, Dec 18, 2021 at 3:38 PM Amit Kapila wrote: > > On Fri, Dec 17, 2021 at 10:51 AM Masahiko Sawada > wrote: > > > > I've attached updated patches. The first patch just moves common > > function for index bulk-deletion and cleanup to vacuum.c. And the > > s

Re: Skipping logical replication transactions on subscriber side

2021-12-17 Thread Masahiko Sawada
c numbers and instead use a syntax that is more > > explicit, like SKIP (xid = NONE) or RESET SKIP or something like that. > > > > +1 for using SKIP (xid = NONE) because otherwise first we need to > introduce RESET syntax for this command. Agreed. Thank you for the comment! Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2021-12-16 Thread Masahiko Sawada
cks - 1) Why do we always need to scan the last page even if heap truncation is disabled (or in the failsafe mode)? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: parallel vacuum comments

2021-12-16 Thread Masahiko Sawada
On Thu, Dec 16, 2021 at 4:27 PM houzj.f...@fujitsu.com wrote: > > On Wed, Dec 15, 2021 4:03 PM Masahiko Sawada wrote: > > On Tue, Dec 14, 2021 at 12:03 PM Amit Kapila > > wrote: > > > There is still pending > > > work related to moving parallel vacuum cod

Re: parallel vacuum comments

2021-12-16 Thread Masahiko Sawada
On Thu, Dec 16, 2021 at 10:38 PM Amit Kapila wrote: > > On Thu, Dec 16, 2021 at 6:13 PM Masahiko Sawada wrote: > > > > On Thu, Dec 16, 2021 at 1:56 PM Amit Kapila wrote: > > > > > > On Wed, Dec 15, 2021 at 1:33 PM Masahiko Sawada > > > wrote: &

Re: parallel vacuum comments

2021-12-16 Thread Masahiko Sawada
On Thu, Dec 16, 2021 at 1:56 PM Amit Kapila wrote: > > On Wed, Dec 15, 2021 at 1:33 PM Masahiko Sawada wrote: > > > > I've attached an updated patch. The patch incorporated several changes > > from the last version: > > > > * Rename parallel_vacuum_begin() t

Re: Skipping logical replication transactions on subscriber side

2021-12-15 Thread Masahiko Sawada
On Thu, Dec 16, 2021 at 2:21 PM Amit Kapila wrote: > > On Thu, Dec 16, 2021 at 10:37 AM Masahiko Sawada > wrote: > > > > On Thu, Dec 16, 2021 at 11:43 AM Amit Kapila > > wrote: > > > > > > I thought we just want to lock before clearing the skip_

Re: Skipping logical replication transactions on subscriber side

2021-12-15 Thread Masahiko Sawada
On Thu, Dec 16, 2021 at 11:43 AM Amit Kapila wrote: > > On Wed, Dec 15, 2021 at 8:19 PM Masahiko Sawada wrote: > > > > On Wed, Dec 15, 2021 at 1:10 PM Amit Kapila wrote: > > > > > > On Wed, Dec 15, 2021 at 8:19 AM Masahiko Sawada > > > wrote: &g

Re: Skipping logical replication transactions on subscriber side

2021-12-15 Thread Masahiko Sawada
On Wed, Dec 15, 2021 at 1:10 PM Amit Kapila wrote: > > On Wed, Dec 15, 2021 at 8:19 AM Masahiko Sawada wrote: > > > > On Tue, Dec 14, 2021 at 2:35 PM Greg Nancarrow wrote: > > > > > > On Tue, Dec 14, 2021 at 3:23 PM vignesh C wrote: > > > > >

Re: parallel vacuum comments

2021-12-15 Thread Masahiko Sawada
On Tue, Dec 14, 2021 at 12:03 PM Amit Kapila wrote: > > On Tue, Dec 14, 2021 at 7:40 AM tanghy.f...@fujitsu.com > wrote: > > > > On Monday, December 13, 2021 2:12 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Dec 13, 2021 at 2:09 PM Amit Kapila

Re: Failed transaction statistics to measure the logical replication progress

2021-12-14 Thread Masahiko Sawada
form only copy operation or > maybe a fixed number of xacts, so, one might not be interested in the > transaction stats of these workers. I find merging only specific stats > of two different types of workers confusing. > > What do others think about this? I understand the concern to have a large number of entries but I agree that merging only specific stats would confuse users. As Amit suggested, it'd be better to show only apply workers' transaction stats. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-12-14 Thread Masahiko Sawada
could happen and seems not good. I thought we can acquire Share lock on pg_subscription during the skip but not sure it's a good idea. It would be better if we can find a way to allow users to specify only XID that has failed. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-12-14 Thread Masahiko Sawada
patch doesn't skip the streaming phase but starts skipping when starting to apply changes. That is, we receive streamed changes and write them to the stream file anyway regardless of skip_xid. When receiving the stream-commit message, we check whether or not we skip this transaction, and if so we apply all messages in the stream file other than all data modification messages. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-12-13 Thread Masahiko Sawada
On Mon, Dec 13, 2021 at 1:04 PM Amit Kapila wrote: > > On Mon, Dec 13, 2021 at 8:28 AM Masahiko Sawada wrote: > > > > On Sat, Dec 11, 2021 at 3:29 PM Amit Kapila wrote: > > > > > > 3. > > > + * Also, we don't skip receiving the changes in streaming c

Re: Assertion failure with replication origins and PREPARE TRANSACTIOn

2021-12-12 Thread Masahiko Sawada
ward. If replication origin LSN and timestamp in commit record and rollback record must be valid data too, I think we should similar checks for commit and rollback code and I think the assertions will fail in the case I reported before[1]. Regards, [1] https://www.postgresql.org/message-

Re: parallel vacuum comments

2021-12-12 Thread Masahiko Sawada
On Mon, Dec 13, 2021 at 2:09 PM Amit Kapila wrote: > > On Mon, Dec 13, 2021 at 10:33 AM Masahiko Sawada > wrote: > > > > On Fri, Dec 10, 2021 at 9:08 PM Amit Kapila wrote: > > > > > > On Thu, Dec 9, 2021 at 6:05 PM Masahiko Sawada > > > wrote

Re: parallel vacuum comments

2021-12-12 Thread Masahiko Sawada
On Fri, Dec 10, 2021 at 9:08 PM Amit Kapila wrote: > > On Thu, Dec 9, 2021 at 6:05 PM Masahiko Sawada wrote: > > > > On Thu, Dec 9, 2021 at 7:44 PM Amit Kapila wrote: > > > > Agreed with the above two points. > > > > I've attached updated patches th

Re: parallel vacuum comments

2021-12-12 Thread Masahiko Sawada
On Mon, Dec 13, 2021 at 12:03 PM Amit Kapila wrote: > > On Sat, Dec 11, 2021 at 8:30 PM Masahiko Sawada wrote: > > > > On Sat, Dec 11, 2021 at 2:32 PM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2021-10-30 14:21:01 -0700, Andre

Re: Skipping logical replication transactions on subscriber side

2021-12-12 Thread Masahiko Sawada
On Sat, Dec 11, 2021 at 3:29 PM Amit Kapila wrote: > > On Fri, Dec 10, 2021 at 11:14 AM Masahiko Sawada > wrote: > > > > On Thu, Dec 9, 2021 at 6:16 PM Amit Kapila wrote: > > > > > > On Thu, Dec 9, 2021 at 2:24 PM Masahiko Sawada > > > wrote:

Re: parallel vacuum comments

2021-12-11 Thread Masahiko Sawada
in vacrel for caller. > > But it actually doesn't know whether we can start workers. It just checks > max_parallel_maintenance_workers, no? Yes, we cannot know whether we can actually start workers when starting parallel index vacuuming. It returns non-NULL if we request one or more workers. Regards -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Replication slot drop message is sent after pgstats shutdown.

2021-12-10 Thread Masahiko Sawada
On Wed, Sep 1, 2021 at 12:37 PM Andres Freund wrote: > > Hi, > > On 2021-09-01 10:05:18 +0900, Masahiko Sawada wrote: > > On Wed, Sep 1, 2021 at 2:39 AM Andres Freund wrote: > > > On 2021-08-31 18:34:12 +0900, Kyotaro Horiguchi wrote: > > > > At Tue, 31 A

Re: Skipping logical replication transactions on subscriber side

2021-12-09 Thread Masahiko Sawada
On Thu, Dec 9, 2021 at 6:16 PM Amit Kapila wrote: > > On Thu, Dec 9, 2021 at 2:24 PM Masahiko Sawada wrote: > > > > On Thu, Dec 9, 2021 at 11:47 AM Amit Kapila wrote: > > > > > > I am thinking that we can start a transaction, update the catalog, > > &

Re: parallel vacuum comments

2021-12-09 Thread Masahiko Sawada
On Thu, Dec 9, 2021 at 7:05 PM Amit Kapila wrote: > > On Mon, Dec 6, 2021 at 10:17 AM Amit Kapila wrote: > > > > On Fri, Dec 3, 2021 at 6:06 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Dec 3, 2021 at 6:03 PM Amit Kapila > > > wr

Re: Skipping logical replication transactions on subscriber side

2021-12-09 Thread Masahiko Sawada
On Thu, Dec 9, 2021 at 11:47 AM Amit Kapila wrote: > > On Wed, Dec 8, 2021 at 4:36 PM Masahiko Sawada wrote: > > > > On Wed, Dec 8, 2021 at 5:54 PM Amit Kapila wrote: > > > > > > On Wed, Dec 8, 2021 at 12:36 PM Masahiko Sawada > > > wrote: > &

Re: Make pg_waldump report replication origin ID, LSN, and timestamp.

2021-12-09 Thread Masahiko Sawada
On Thu, Dec 9, 2021 at 4:02 PM Michael Paquier wrote: > > On Wed, Dec 08, 2021 at 05:03:30PM +0900, Masahiko Sawada wrote: > > Agreed. I've attached an updated patch that incorporated your review > > comments. Please review it. > > That looks correct to me. One thing t

Re: cutting down the TODO list thread

2021-12-08 Thread Masahiko Sawada
rS91Nq0eVQ0B3BA%40mail.gmail.com But not completed. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-12-08 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 4:05 PM Masahiko Sawada wrote: > > Okay, I understand those cases but note always checking if the > > prepared xact exists during commit prepared has a cost and that is why > > we avoided it at the first place. BTW what costs were we concerned about? Look

Re: Skipping logical replication transactions on subscriber side

2021-12-08 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 5:54 PM Amit Kapila wrote: > > On Wed, Dec 8, 2021 at 12:36 PM Masahiko Sawada wrote: > > > > On Wed, Dec 8, 2021 at 3:50 PM Amit Kapila wrote: > > > > > > On Wed, Dec 8, 2021 at 11:48 AM Masahiko Sawada > > > wrote: >

Re: Make pg_waldump report replication origin ID, LSN, and timestamp.

2021-12-08 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 4:31 PM Michael Paquier wrote: > > On Mon, Dec 06, 2021 at 11:24:09PM +0900, Masahiko Sawada wrote: > > On Mon, Dec 6, 2021 at 5:09 PM Michael Paquier wrote: > >> Shouldn't you check for parsed.origin_lsn instead? The replication > >> orig

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 3:50 PM Amit Kapila wrote: > > On Wed, Dec 8, 2021 at 11:48 AM Masahiko Sawada wrote: > > > > On Wed, Dec 8, 2021 at 2:16 PM Amit Kapila wrote: > > > > > > On Tue, Dec 7, 2021 at 5:06 PM Masahiko Sawada > > > wrote: > &

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 2:16 PM Amit Kapila wrote: > > On Tue, Dec 7, 2021 at 5:06 PM Masahiko Sawada wrote: > > > > On Mon, Dec 6, 2021 at 2:17 PM Amit Kapila wrote: > > > > I'll submit the patch tomorrow. > > > > While updating the pat

Re: parallel vacuum comments

2021-12-07 Thread Masahiko Sawada
On Wed, Dec 8, 2021 at 12:22 PM houzj.f...@fujitsu.com wrote: > > On Tuesday, December 7, 2021 1:42 PM Masahiko Sawada > wrote: > > I've attached an updated patch. I've removed 0003 patch that added > > regression tests as per discussion. Regarding the term

Re: Skipping logical replication transactions on subscriber side

2021-12-07 Thread Masahiko Sawada
On Mon, Dec 6, 2021 at 2:17 PM Amit Kapila wrote: > > On Fri, Dec 3, 2021 at 12:12 PM Masahiko Sawada wrote: > > > > On Fri, Dec 3, 2021 at 11:53 AM Amit Kapila wrote: > > > > > > > But this syntax gives you flexibility, so we can also >

Re: parallel vacuum comments

2021-12-06 Thread Masahiko Sawada
On Mon, Dec 6, 2021 at 1:47 PM Amit Kapila wrote: > > On Fri, Dec 3, 2021 at 6:06 PM Masahiko Sawada wrote: > > > > On Fri, Dec 3, 2021 at 6:03 PM Amit Kapila wrote: > > > > > > On Thu, Dec 2, 2021 at 6:01 PM Masahiko Sawada > > > wrote:

Re: Make pg_waldump report replication origin ID, LSN, and timestamp.

2021-12-06 Thread Masahiko Sawada
On Mon, Dec 6, 2021 at 11:24 PM Masahiko Sawada wrote: > > On Mon, Dec 6, 2021 at 5:09 PM Michael Paquier wrote: > > > > On Mon, Dec 06, 2021 at 04:35:07PM +0900, Masahiko Sawada wrote: > > > I've attached a patch to add replication origin information to > > &g

Fix a bug in DecodeAbort() and improve input data check on subscriber.

2021-12-06 Thread Masahiko Sawada
ome new checks. Since it could be frequently called, I used unlikely() but probably we can also consider replacing elog(ERROR) with assertions. Regards, [1] https://www.postgresql.org/message-id/CAD21AoD2dJfgsdxk4_KciAZMZQoUiCvmV9sDpp8ZuKLtKCNXaA%40mail.gmail.com -- Masahiko Sawada EDB: http

Re: Make pg_waldump report replication origin ID, LSN, and timestamp.

2021-12-06 Thread Masahiko Sawada
On Mon, Dec 6, 2021 at 5:09 PM Michael Paquier wrote: > > On Mon, Dec 06, 2021 at 04:35:07PM +0900, Masahiko Sawada wrote: > > I've attached a patch to add replication origin information to > > xact_desc_prepare(). > > Yeah. > > + if (origin_id != InvalidRepOrigin

Make pg_waldump report replication origin ID, LSN, and timestamp.

2021-12-05 Thread Masahiko Sawada
/message-id/CAHGQGwEvhASad4JJnCv%3D0dW2TJypZgW_Vpb-oZik2a3utCqcrA%40mail.gmail.com -- Masahiko Sawada EDB: https://www.enterprisedb.com/ From b37d5a8cd5b84eeb850bd7ddeb903024652f20d2 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada Date: Mon, 6 Dec 2021 14:32:57 +0900 Subject: [PATCH] Make

Re: parallel vacuum comments

2021-12-03 Thread Masahiko Sawada
On Fri, Dec 3, 2021 at 6:03 PM Amit Kapila wrote: > > On Thu, Dec 2, 2021 at 6:01 PM Masahiko Sawada wrote: > > > > I've attached updated patches. > > > > I have a few comments on v4-0001. Thank you for the comments! > 1. > In parallel_vacuum_process_a

Re: parallel vacuum comments

2021-12-03 Thread Masahiko Sawada
and On Fri, Dec 3, 2021 at 6:56 PM Amit Kapila wrote: > > On Fri, Dec 3, 2021 at 2:33 PM Amit Kapila wrote: > > > > On Thu, Dec 2, 2021 at 6:01 PM Masahiko Sawada > > wrote: > > > > > > I've attached updated patches. > > > > > &

Re: Skipping logical replication transactions on subscriber side

2021-12-02 Thread Masahiko Sawada
to achieve. If we allow specifying operations and/or relations, probably multiple operations or relations need to be specified in some cases. Otherwise, the subscriber cannot continue logical replication if the transaction has multiple operations on different relations that fail. But similar to the idea

Re: Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-30 Thread Masahiko Sawada
On Wed, Dec 1, 2021 at 4:42 AM Peter Geoghegan wrote: > > On Mon, Nov 29, 2021 at 7:00 PM Masahiko Sawada wrote: > > Thanks! I'll change my parallel vacuum refactoring patch accordingly. > > Thanks again for working on that. > > > Regarding the commit, I think th

Re: Skipping logical replication transactions on subscriber side

2021-11-30 Thread Masahiko Sawada
On Wed, Dec 1, 2021 at 1:00 PM Amit Kapila wrote: > > On Wed, Dec 1, 2021 at 9:12 AM Masahiko Sawada wrote: > > > > On Wed, Dec 1, 2021 at 12:22 PM Amit Kapila wrote: > > > > > > On Wed, Dec 1, 2021 at 8:24 AM houzj.f...@fujitsu.com > > > w

Re: Skipping logical replication transactions on subscriber side

2021-11-30 Thread Masahiko Sawada
On Wed, Dec 1, 2021 at 12:22 PM Amit Kapila wrote: > > On Wed, Dec 1, 2021 at 8:24 AM houzj.f...@fujitsu.com > wrote: > > > > On Tues, Nov 30, 2021 9:39 PM Masahiko Sawada wrote: > > > > > > > > > Shouldn't we someway check that the error message

Re: Skipping logical replication transactions on subscriber side

2021-11-30 Thread Masahiko Sawada
On Tue, Nov 30, 2021 at 8:41 PM Masahiko Sawada wrote: > > On Tue, Nov 30, 2021 at 6:28 PM Amit Kapila wrote: > > > > On Mon, Nov 29, 2021 at 11:38 AM vignesh C wrote: > > > > > > > I have pushed this patch and there is a buildfarm failure for it. See: &

Re: Skipping logical replication transactions on subscriber side

2021-11-30 Thread Masahiko Sawada
st_subscription_error() which is not what we expect in the test? Right. > Shouldn't we someway check that the error message also starts with > "duplicate key value violates ..."? Yeah, I think it's a good idea to make the checks more specific. That is, probably we can specify the prefix of the error message and subrelid in addition to the current conditions: relid and xid. That way, we can check what error was reported by which workers (tablesync or apply) for which relations. And both check queries in test_subscription_error() can have the same WHERE clause. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: parallel vacuum comments

2021-11-29 Thread Masahiko Sawada
On Tue, Nov 30, 2021 at 3:00 PM Amit Kapila wrote: > > On Tue, Nov 30, 2021 at 11:03 AM houzj.f...@fujitsu.com > wrote: > > > > On Mon, Nov 29, 2021 11:38 AM Masahiko Sawada wrote: > > > > > > > 2) > > + /* Reinitialize the pa

Re: Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-29 Thread Masahiko Sawada
On Tue, Nov 30, 2021 at 3:00 AM Peter Geoghegan wrote: > > On Wed, Nov 24, 2021 at 4:48 AM Masahiko Sawada wrote: > > The patch renames dead tuples to dead items at some places and to > > dead TIDs at some places. > > > I think it's more consistent if we change it

Re: parallel vacuum comments

2021-11-28 Thread Masahiko Sawada
On Wed, Nov 24, 2021 at 5:54 PM Amit Kapila wrote: > > On Wed, Nov 24, 2021 at 12:16 PM Masahiko Sawada > wrote: > > > > On Wed, Nov 24, 2021 at 1:34 PM Amit Kapila wrote: > > > > > > On Wed, Nov 24, 2021 at 7:43 AM Masahiko Sawada > > > wrote:

Re: Skipping logical replication transactions on subscriber side

2021-11-28 Thread Masahiko Sawada
On Sat, Nov 27, 2021 at 7:56 PM Amit Kapila wrote: > > On Fri, Nov 26, 2021 at 6:00 AM Masahiko Sawada wrote: > > > > Indeed. Attached an updated patch. Thanks! > > > Thank you for updating the patch! > I have made a number of changes in the attached patch

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 10:06 PM houzj.f...@fujitsu.com wrote: > > On Thur, Nov 25, 2021 8:29 PM Masahiko Sawada wrote: > > On Thu, Nov 25, 2021 at 1:57 PM Amit Kapila wrote: > > > > > > On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada > > wrote: &

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 9:08 PM Greg Nancarrow wrote: > > On Wed, Nov 24, 2021 at 10:44 PM Masahiko Sawada > wrote: > > > > I've attached an updated version patch. Unless I miss something, all > > comments I got so far have been incorporated into this patch. Please

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 7:36 PM vignesh C wrote: > > On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada wrote: > > > > On Wed, Nov 17, 2021 at 8:14 PM Amit Kapila wrote: > > > > > > On Tue, Nov 16, 2021 at 12:01 PM Masahiko Sawada > > > wrote:

Re: Skipping logical replication transactions on subscriber side

2021-11-25 Thread Masahiko Sawada
On Thu, Nov 25, 2021 at 1:57 PM Amit Kapila wrote: > > On Wed, Nov 24, 2021 at 5:14 PM Masahiko Sawada wrote: > > > > Changed. I've removed first_error_time as per discussion on the thread > > for adding xact stats. > > > > We also agreed to change the co

Re: Rename dead_tuples to dead_items in vacuumlazy.c

2021-11-24 Thread Masahiko Sawada
space used * for dead_items. */ Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-11-24 Thread Masahiko Sawada
On Wed, Nov 17, 2021 at 8:14 PM Amit Kapila wrote: > > On Tue, Nov 16, 2021 at 12:01 PM Masahiko Sawada > wrote: > > > > Right. I've fixed this issue and attached an updated patch. > > > > Few comments/questions: > = > 1. > + &

Re: Skipping logical replication transactions on subscriber side

2021-11-24 Thread Masahiko Sawada
On Wed, Nov 24, 2021 at 12:13 PM Amit Kapila wrote: > > On Wed, Nov 24, 2021 at 7:51 AM Masahiko Sawada wrote: > > > > On Thu, Nov 18, 2021 at 12:52 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Tuesday, November 16, 2021 2:31 PM Masahiko Sawad

Re: parallel vacuum comments

2021-11-23 Thread Masahiko Sawada
On Wed, Nov 24, 2021 at 1:34 PM Amit Kapila wrote: > > On Wed, Nov 24, 2021 at 7:43 AM Masahiko Sawada wrote: > > > > On Mon, Nov 22, 2021 at 1:48 PM Amit Kapila wrote: > > > > > > On Fri, Nov 19, 2021 at 7:55 AM houzj.f...@fujitsu.com > > > wrote:

Re: parallel vacuum comments

2021-11-23 Thread Masahiko Sawada
On Wed, Nov 24, 2021 at 1:28 PM Amit Kapila wrote: > > On Wed, Nov 24, 2021 at 7:07 AM Masahiko Sawada wrote: > > > > On Mon, Nov 22, 2021 at 6:35 PM Amit Kapila wrote: > > > > > > On Tue, Nov 16, 2021 at 11:23 AM Masahiko Sawada > > > wrote: >

Re: Synchronizing slots from primary to standby

2021-11-23 Thread Masahiko Sawada
he slot name to sync with the primary. I'm not sure we already discussed this topic but I think we need it at least for testing purposes. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-11-23 Thread Masahiko Sawada
On Thu, Nov 18, 2021 at 12:52 PM houzj.f...@fujitsu.com wrote: > > On Tuesday, November 16, 2021 2:31 PM Masahiko Sawada > wrote: > > Right. I've fixed this issue and attached an updated patch. > > > Hi, > > I have few comments for the testcases. >

Re: parallel vacuum comments

2021-11-23 Thread Masahiko Sawada
On Fri, Nov 19, 2021 at 11:25 AM houzj.f...@fujitsu.com wrote: > > On Tues, Nov 16, 2021 1:53 PM Masahiko Sawada wrote: > > I've incorporated these comments and attached an updated patch. > > Thanks for updating the patch. > I read the latest patch and have few

Re: parallel vacuum comments

2021-11-23 Thread Masahiko Sawada
On Mon, Nov 22, 2021 at 1:48 PM Amit Kapila wrote: > > On Fri, Nov 19, 2021 at 7:55 AM houzj.f...@fujitsu.com > wrote: > > > > On Tues, Nov 16, 2021 1:53 PM Masahiko Sawada wrote: > > > I've incorporated these comments and attached an updated patch. &

Re: parallel vacuum comments

2021-11-23 Thread Masahiko Sawada
On Mon, Nov 22, 2021 at 6:35 PM Amit Kapila wrote: > > On Tue, Nov 16, 2021 at 11:23 AM Masahiko Sawada > wrote: > > > > I've incorporated these comments and attached an updated patch. > > > > Review comments: > > 1.

Re: Failed transaction statistics to measure the logical replication progress

2021-11-23 Thread Masahiko Sawada
On Tue, Nov 23, 2021 at 3:21 PM Greg Nancarrow wrote: > > On Sat, Nov 20, 2021 at 1:11 AM Masahiko Sawada wrote: > > > > I'm concerned that these new names will introduce confusion; if we > > have last_error_relid, last_error_command, last_error_message, > > last_

Re: Failed transaction statistics to measure the logical replication progress

2021-11-23 Thread Masahiko Sawada
On Sat, Nov 20, 2021 at 3:25 PM Amit Kapila wrote: > > On Fri, Nov 19, 2021 at 7:41 PM Masahiko Sawada wrote: > > > > On Thu, Nov 18, 2021 at 12:26 PM Amit Kapila > > wrote: > > > > > > On Wed, Nov 17, 2021 at 7:12 PM osumi.takami...@fujitsu.com

Re: Failed transaction statistics to measure the logical replication progress

2021-11-19 Thread Masahiko Sawada
account by ReorderBufferChangeSize(). I guess that the purpose of these values is to compare them to total_bytes, stream_byte, and spill_bytes but if the calculation is not accurate, does it mean that the more stats are updated, the more the stats will be getting inaccurate? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-11-19 Thread Masahiko Sawada
se this problem, the next message will also be reported soon, fixing it soon, as Amit mentioned. Also, IIUC once we have the shared memory based stats collector, we won’t need to worry about this problem. Given that this kind of problem potentially exists also in other stats views that have timestamp

Re: Skipping logical replication transactions on subscriber side

2021-11-18 Thread Masahiko Sawada
On Wed, Nov 17, 2021 at 7:46 PM vignesh C wrote: > > On Tue, Nov 16, 2021 at 12:01 PM Masahiko Sawada > wrote: > > > > On Mon, Nov 15, 2021 at 11:43 PM vignesh C wrote: > > > > > > On Mon, Nov 15, 2021 at 2:48 PM Masahiko Sawada > > > wrote:

Re: Skipping logical replication transactions on subscriber side

2021-11-18 Thread Masahiko Sawada
On Wed, Nov 17, 2021 at 12:43 PM houzj.f...@fujitsu.com wrote: > > On Tues, Nov 16, 2021 2:31 PM Masahiko Sawada wrote: > > Right. I've fixed this issue and attached an updated patch. > > Hi, > > Thanks for updating the patch. > Here are few comments. Thank y

Re: Skipping logical replication transactions on subscriber side

2021-11-18 Thread Masahiko Sawada
On Thu, Nov 18, 2021 at 5:45 PM tanghy.f...@fujitsu.com wrote: > > On Tuesday, November 16, 2021 2:31 PM Masahiko Sawada > wrote: > > > > Right. I've fixed this issue and attached an updated patch. > > > > > > Thanks for your patch. > > I read the

Re: CREATE PUBLICATION should "See Also" CREATE SUBSCRIPTION

2021-11-17 Thread Masahiko Sawada
docs page > > [2]. > > Nice catch. > > > > > > > > > + > > To make it analogous with how CREATE/ALTER for SUBSCRIPTION and PUBLICATION > reference each other, there should IMO be another xref to ALTER SUBSCRIPTION > as > well. +1 Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Skipping logical replication transactions on subscriber side

2021-11-16 Thread Masahiko Sawada
On Wed, Nov 17, 2021 at 3:52 PM vignesh C wrote: > > On Tue, Nov 16, 2021 at 12:01 PM Masahiko Sawada > wrote: > > > > On Mon, Nov 15, 2021 at 11:43 PM vignesh C wrote: > > > > > > On Mon, Nov 15, 2021 at 2:48 PM Masahiko Sawada > > > wrote:

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-16 Thread Masahiko Sawada
On Tue, Nov 16, 2021 at 8:45 PM Amit Kapila wrote: > > On Mon, Nov 15, 2021 at 12:38 PM Masahiko Sawada > wrote: > > > > I've updated the patch so that ProcArrayInstallRestoredXmin() sets > > both xmin and statusFlags only when the source proc is still running > &

Re: Skipping logical replication transactions on subscriber side

2021-11-16 Thread Masahiko Sawada
On Wed, Nov 17, 2021 at 1:58 PM Amit Kapila wrote: > > On Wed, Nov 17, 2021 at 9:13 AM houzj.f...@fujitsu.com > wrote: > > > > On Tues, Nov 16, 2021 2:31 PM Masahiko Sawada wrote: > > > > 2) > > + > > + > > + subrelid oid >

Re: Failed transaction statistics to measure the logical replication progress

2021-11-16 Thread Masahiko Sawada
t introduces pg_stat_subscription_workers view? --- I think that exporting PartitionTupleRouting should not be done in the one patch together with renaming the view columns. There is not relevance between them at all. If it's used by v12-0002 patch, I think it should be included in that patch or in another separate patch. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Showing I/O timings spent reading/writing temp buffers in EXPLAIN

2021-11-15 Thread Masahiko Sawada
On Tue, Sep 7, 2021 at 12:02 AM wrote: > > Hi, > > ‐‐‐ Original Message ‐‐‐ > On Tuesday, August 24th, 2021 at 13:20, Ranier Vilela > wrote: > > > Em ter., 24 de ago. de 2021 às 03:11, Masahiko Sawada > > escreveu: > > > > >

Re: Skipping logical replication transactions on subscriber side

2021-11-15 Thread Masahiko Sawada
On Mon, Nov 15, 2021 at 11:43 PM vignesh C wrote: > > On Mon, Nov 15, 2021 at 2:48 PM Masahiko Sawada wrote: > > > > On Mon, Nov 15, 2021 at 4:49 PM Greg Nancarrow wrote: > > > > > > On Mon, Nov 15, 2021 at 1:49 PM Masahiko Sawada > > > wrote: &

Re: parallel vacuum comments

2021-11-15 Thread Masahiko Sawada
On Tue, Nov 16, 2021 at 11:38 AM houzj.f...@fujitsu.com wrote: > > On Thur, Nov 11, 2021 10:41 AM Masahiko Sawada wrote: > > I've attached a draft patch that refactors parallel vacuum and separates > > parallel-vacuum-related code to new file vacuumparallel.c. > > After

Re: parallel vacuum comments

2021-11-15 Thread Masahiko Sawada
On Mon, Nov 15, 2021 at 8:04 PM Amit Kapila wrote: > > On Mon, Nov 15, 2021 at 2:01 PM Masahiko Sawada wrote: > > > > On Thu, Nov 11, 2021 at 6:38 PM Amit Kapila wrote: > > > > > > On Thu, Nov 11, 2021 at 8:11 AM Masahiko Sawada > > > wrote

Re: Skipping logical replication transactions on subscriber side

2021-11-15 Thread Masahiko Sawada
On Mon, Nov 15, 2021 at 4:49 PM Greg Nancarrow wrote: > > On Mon, Nov 15, 2021 at 1:49 PM Masahiko Sawada wrote: > > > > I've attached an updated patch that incorporates all comments I got so > > far. Please review it. > > > > Thanks for the updated patch.

Re: parallel vacuum comments

2021-11-15 Thread Masahiko Sawada
On Thu, Nov 11, 2021 at 6:38 PM Amit Kapila wrote: > > On Thu, Nov 11, 2021 at 8:11 AM Masahiko Sawada wrote: > > > > I've attached a draft patch that refactors parallel vacuum and > > separates parallel-vacuum-related code to new file vacuumparallel.c. > >

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-14 Thread Masahiko Sawada
On Sat, Nov 13, 2021 at 2:10 PM Amit Kapila wrote: > > On Fri, Nov 12, 2021 at 6:44 PM Alvaro Herrera > wrote: > > > > On 2021-Nov-11, Masahiko Sawada wrote: > > > > > On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila > > > wrote: > > > &

Re: Skipping logical replication transactions on subscriber side

2021-11-14 Thread Masahiko Sawada
aster/postmaster.h" > +#include "replication/logicalproto.h" > #include "replication/slot.h" > #include "storage/proc.h" Removed; > > 7) There is an extra ";", We can remove one ";" from below: > + PgStat_StatSubWorkerKey key; > + boolfound; > + HASHACTION action = (create ? HASH_ENTER : HASH_FIND);; > + > + key.subid = subid; > + key.subrelid = subrelid; Fixed. I've attached an updated patch that incorporates all comments I got so far. Please review it. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ v21-0001-Add-a-subscription-worker-statistics-view-pg_sta.patch Description: Binary data

Re: Skipping logical replication transactions on subscriber side

2021-11-14 Thread Masahiko Sawada
On Tue, Nov 9, 2021 at 3:07 PM Dilip Kumar wrote: > > On Sun, Nov 7, 2021 at 7:50 PM Masahiko Sawada wrote: > > I've attached an updated patch. In this version patch, subscription > > worker statistics are collected per-database and handled in a similar > > way to table

Re: Skipping logical replication transactions on subscriber side

2021-11-14 Thread Masahiko Sawada
On Mon, Nov 8, 2021 at 4:10 PM Greg Nancarrow wrote: > > On Mon, Nov 8, 2021 at 1:20 AM Masahiko Sawada wrote: > > > > I've attached an updated patch. In this version patch, subscription > > worker statistics are collected per-database and handled in a similar > >

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-11 Thread Masahiko Sawada
On Thu, Nov 11, 2021 at 3:07 PM Amit Kapila wrote: > > On Thu, Nov 11, 2021 at 10:40 AM Masahiko Sawada > wrote: > > > > On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila > > wrote: > > > > > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: >

Re: Improve logging when using Huge Pages

2021-11-10 Thread Masahiko Sawada
case, it’d be better to log the message on failure with the request memory size (or whatever reason for the failure). That is, we end up logging such a message on failure when huge_pages = on/try. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Masahiko Sawada
On Thu, Nov 11, 2021 at 12:53 PM Amit Kapila wrote: > > On Thu, Nov 11, 2021 at 9:11 AM Andres Freund wrote: > > > > Hi, > > > > On 2021-11-11 12:22:42 +0900, Masahiko Sawada wrote: > > > > 2. > > > > LWLockAcquire(ProcArrayLock, L

Re: Parallel vacuum workers prevent the oldest xmin from advancing

2021-11-10 Thread Masahiko Sawada
On Wed, Nov 10, 2021 at 6:14 PM Amit Kapila wrote: > > On Fri, Oct 22, 2021 at 11:08 AM Masahiko Sawada > wrote: > > > > On Wed, Oct 20, 2021 at 9:27 AM Masahiko Sawada > > wrote: > > > > I agree to copy statusFlags in ProcArrayInstallRestoredXmin().

Re: parallel vacuum comments

2021-11-10 Thread Masahiko Sawada
On Tue, Nov 9, 2021 at 9:53 PM Masahiko Sawada wrote: > > On Fri, Nov 5, 2021 at 4:00 AM Andres Freund wrote: > > > > Hi, > > > > On 2021-11-01 10:44:34 +0900, Masahiko Sawada wrote: > > > On Sun, Oct 31, 2021 at 6:21 AM Andres Freund wrote: >

<    6   7   8   9   10   11   12   13   14   15   >