Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-21 Thread Masahiko Sawada
On Tue, Jul 19, 2022 at 1:30 PM John Naylor wrote: > > > > On Tue, Jul 19, 2022 at 9:11 AM Masahiko Sawada wrote: > > > I’d like to keep the first version simple. We can improve it and add > > more optimizations later. Using radix tree for vacuum TID storage &

Re: Improve description of XLOG_RUNNING_XACTS

2022-07-21 Thread Masahiko Sawada
On Thu, Jul 21, 2022 at 4:29 PM Kyotaro Horiguchi wrote: > > At Thu, 21 Jul 2022 11:21:09 +0900, Fujii Masao > wrote in > > > > > > On 2022/07/21 10:13, Masahiko Sawada wrote: > > > Hi, > > > I realized that standby_desc_running_xacts() in standb

Improve description of XLOG_RUNNING_XACTS

2022-07-20 Thread Masahiko Sawada
subxacts: 1049 Please review it. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ 0001-Improve-description-of-XLOG_RUNNING_XACTS.patch Description: Binary data

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-20 Thread Masahiko Sawada
On Wed, Jul 20, 2022 at 2:19 PM Amit Kapila wrote: > > On Wed, Jul 20, 2022 at 9:01 AM Masahiko Sawada wrote: > > > > On Wed, Jul 20, 2022 at 12:11 PM Amit Kapila > > wrote: > > > > > > On Tue, Jul 19, 2022 at 7:28 PM Masahiko Sawada > > >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-20 Thread Masahiko Sawada
On Wed, Jul 20, 2022 at 4:16 PM Kyotaro Horiguchi wrote: > > At Wed, 20 Jul 2022 10:58:16 +0900, Masahiko Sawada > wrote in > > On Wed, Jul 20, 2022 at 9:58 AM Kyotaro Horiguchi > > wrote: > > > Mmm. the patch changed that behavior. AllocateSnapshotBuilder a

Re: Transparent column encryption

2022-07-20 Thread Masahiko Sawada
+ PG_AD_LEN + encrlen) = pg_hton64(PG_AD_LEN * 8); 318 319 if (!EVP_DigestSignInit(evp_md_ctx, NULL, md, NULL, pkey)) (lldb) p encrlen (int) $0 = -12 (lldb) Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-19 Thread Masahiko Sawada
On Wed, Jul 20, 2022 at 12:11 PM Amit Kapila wrote: > > On Tue, Jul 19, 2022 at 7:28 PM Masahiko Sawada wrote: > > > > On Tue, Jul 19, 2022 at 9:25 PM Amit Kapila wrote: > > > > > > On Tue, Jul 19, 2022 at 1:10 PM Masahiko Sawada > > > wrote: >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-19 Thread Masahiko Sawada
On Wed, Jul 20, 2022 at 9:58 AM Kyotaro Horiguchi wrote: > > At Tue, 19 Jul 2022 17:31:07 +0900, Masahiko Sawada > wrote in > > On Tue, Jul 19, 2022 at 4:35 PM Kyotaro Horiguchi > > wrote: > > > At Tue, 19 Jul 2022 10:17:15 +0530, Amit Kapila > > &g

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-19 Thread Masahiko Sawada
On Tue, Jul 19, 2022 at 9:25 PM Amit Kapila wrote: > > On Tue, Jul 19, 2022 at 1:10 PM Masahiko Sawada wrote: > > > > On Mon, Jul 18, 2022 at 8:49 PM Amit Kapila wrote: > > > > > > On Sun, Jul 17, 2022 at 6:29 PM Masahiko Sawada > > > wrote: &g

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-19 Thread Masahiko Sawada
ed here. The > assert just requires that catchange_txns and catchange_ntxns are > consistent so it should be checked just after dlist_empty.. I think. > If we want to check if catchange_txns and catchange_ntxns are consistent, should we check (xcnt == rb->catchange_ntxns) as well, no? This function requires the caller to use rb->catchange_ntxns as the length of the returned array. I think this assertion ensures that the actual length of the array is consistent with the length we pre-calculated. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-19 Thread Masahiko Sawada
On Tue, Jul 19, 2022 at 4:28 PM osumi.takami...@fujitsu.com wrote: > > On Sunday, July 17, 2022 9:59 PM Masahiko Sawada > wrote: > > I've attached patches for all supported branches including the master. > Hi, > > > Minor comments for REL14. > > (1)

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-19 Thread Masahiko Sawada
On Mon, Jul 18, 2022 at 8:49 PM Amit Kapila wrote: > > On Sun, Jul 17, 2022 at 6:29 PM Masahiko Sawada wrote: > > > > On Fri, Jul 15, 2022 at 3:32 PM shiy.f...@fujitsu.com > > wrote: > > > > > > > I've attached patches for all supported branches

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-19 Thread Masahiko Sawada
On Tue, Jul 19, 2022 at 1:47 PM Amit Kapila wrote: > > On Tue, Jul 19, 2022 at 6:34 AM Masahiko Sawada wrote: > > > > On Mon, Jul 18, 2022 at 1:12 PM Amit Kapila wrote: > > > > > > On Fri, Jul 15, 2022 at 8:09 PM Masahiko Sawada > > > wrote: &

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-18 Thread Masahiko Sawada
On Thu, Jul 14, 2022 at 1:17 PM John Naylor wrote: > > On Tue, Jul 12, 2022 at 8:16 AM Masahiko Sawada wrote: > > > > > I think that at this stage it's better to define the design first. For > > > > example, key size and value size, and these sizes are fixed o

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-18 Thread Masahiko Sawada
On Mon, Jul 18, 2022 at 12:28 PM shiy.f...@fujitsu.com wrote: > > On Fri, Jul 15, 2022 10:39 PM Masahiko Sawada wrote: > > > > This patch should have the fix for the issue that Shi yu reported. Shi > > yu, could you please test it again with this patch? > > > >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-18 Thread Masahiko Sawada
On Mon, Jul 18, 2022 at 1:12 PM Amit Kapila wrote: > > On Fri, Jul 15, 2022 at 8:09 PM Masahiko Sawada wrote: > > > > This patch should have the fix for the issue that Shi yu reported. Shi > > yu, could you please test it again with this patch? > > > > Can

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-17 Thread Masahiko Sawada
On Fri, Jul 15, 2022 at 3:32 PM shiy.f...@fujitsu.com wrote: > > On Mon, Jul 11, 2022 9:54 PM Masahiko Sawada wrote: > > > > I've attached an updated patch, please review it. > > > > Thanks for your patch. Here are some comments for the REL14-v1 patch. > > 1.

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-15 Thread Masahiko Sawada
On Fri, Jul 15, 2022 at 10:43 PM osumi.takami...@fujitsu.com wrote: > > On Thursday, July 14, 2022 10:31 AM Masahiko Sawada > wrote: > > I've attached an updated patch that incorporated comments from Amit and Shi. > Hi, > > > Minor comments for v4. Thank you for

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-14 Thread Masahiko Sawada
On Thu, Jul 14, 2022 at 12:06 PM Masahiko Sawada wrote: > > On Thu, Jul 14, 2022 at 11:16 AM shiy.f...@fujitsu.com > wrote: > > > > On Tue, Jul 12, 2022 5:23 PM Masahiko Sawada wrote: > > > > > > On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com &g

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-13 Thread Masahiko Sawada
On Thu, Jul 14, 2022 at 11:16 AM shiy.f...@fujitsu.com wrote: > > On Tue, Jul 12, 2022 5:23 PM Masahiko Sawada wrote: > > > > On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com > > wrote: > > > > > > It happened when executing the followin

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-13 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 12:40 PM shiy.f...@fujitsu.com wrote: > > On Tue, Jul 12, 2022 8:49 AM Masahiko Sawada wrote: > > > > I've attached an updated patch. > > > > While trying this idea, I noticed there is no API to get the length of > > dlist, as we

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-13 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 5:52 PM Amit Kapila wrote: > > On Tue, Jul 12, 2022 at 1:13 PM Masahiko Sawada wrote: > > > > On Tue, Jul 12, 2022 at 3:25 PM Amit Kapila wrote: > > > > > > On Tue, Jul 12, 2022 at 11:38 AM Masahiko Sawada > > > wrote: >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 7:59 PM Amit Kapila wrote: > > On Tue, Jul 12, 2022 at 2:53 PM Masahiko Sawada wrote: > > > > On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com > > wrote: > > > > > > > > > It happened when executing the fo

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 5:58 PM shiy.f...@fujitsu.com wrote: > > On Tue, Jul 12, 2022 8:49 AM Masahiko Sawada wrote: > > > > I've attached an updated patch. > > > > Hi, > > I met a segmentation fault in test_decoding test after applying the patch for >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 3:25 PM Amit Kapila wrote: > > On Tue, Jul 12, 2022 at 11:38 AM Masahiko Sawada > wrote: > > > > On Tue, Jul 12, 2022 at 10:28 AM Masahiko Sawada > > wrote: > > > > > > > > > I'm doing benchmark tests and will shar

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-12 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 10:28 AM Masahiko Sawada wrote: > > On Tue, Jul 12, 2022 at 9:48 AM Masahiko Sawada wrote: > > > > On Fri, Jul 8, 2022 at 8:20 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Jul 8, 2022 at 5:59 PM Amit Kapila > > &g

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-11 Thread Masahiko Sawada
On Tue, Jul 12, 2022 at 9:48 AM Masahiko Sawada wrote: > > On Fri, Jul 8, 2022 at 8:20 PM Masahiko Sawada wrote: > > > > On Fri, Jul 8, 2022 at 5:59 PM Amit Kapila wrote: > > > > > > On Fri, Jul 8, 2022 at 12:46 PM Masahiko Sawada > > > wrote: &

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-11 Thread Masahiko Sawada
On Fri, Jul 8, 2022 at 3:43 PM John Naylor wrote: > > On Fri, Jul 8, 2022 at 9:10 AM Masahiko Sawada wrote: > > > I guess that the tree height is affected by where garbages are, right? > > For example, even if all garbage in the table is concentrated in > > 0.5GB,

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-11 Thread Masahiko Sawada
On Fri, Jul 8, 2022 at 8:20 PM Masahiko Sawada wrote: > > On Fri, Jul 8, 2022 at 5:59 PM Amit Kapila wrote: > > > > On Fri, Jul 8, 2022 at 12:46 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Jul 8, 2022 at 3:27 PM Amit Kap

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-11 Thread Masahiko Sawada
On Wed, Jul 6, 2022 at 3:01 PM Amit Kapila wrote: > > On Wed, Jul 6, 2022 at 7:38 AM Masahiko Sawada wrote: > > > > I'll post a new version patch in the next email with replying to other > > comments. > > > > Okay, thanks for worki

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-08 Thread Masahiko Sawada
On Fri, Jul 8, 2022 at 5:59 PM Amit Kapila wrote: > > On Fri, Jul 8, 2022 at 12:46 PM Masahiko Sawada wrote: > > > > On Fri, Jul 8, 2022 at 3:27 PM Amit Kapila wrote: > > > > > > > > 1. > > > In ReorderBufferGetCatalogChangesXa

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-08 Thread Masahiko Sawada
On Fri, Jul 8, 2022 at 3:27 PM Amit Kapila wrote: > > On Fri, Jul 8, 2022 at 6:45 AM Masahiko Sawada wrote: > > > > On Thu, Jul 7, 2022 at 3:40 PM Amit Kapila wrote: > > > > > > On Thu, Jul 7, 2022 at 8:21 AM Masahiko Sawada > > > wrote

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-07 Thread Masahiko Sawada
On Tue, Jul 5, 2022 at 5:49 PM John Naylor wrote: > > On Mon, Jul 4, 2022 at 12:07 PM Masahiko Sawada wrote: > > > > Looking at the node stats, and then your benchmark code, I think key > > > construction is a major influence, maybe more than node type. The > &

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-07 Thread Masahiko Sawada
On Thu, Jul 7, 2022 at 3:40 PM Amit Kapila wrote: > > On Thu, Jul 7, 2022 at 8:21 AM Masahiko Sawada wrote: > > > > On Wed, Jul 6, 2022 at 5:55 PM Amit Kapila wrote: > > > > > > On Wed, Jul 6, 2022 at 12:19 PM Masahiko Sawada > > > wrote: > &

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-06 Thread Masahiko Sawada
On Wed, Jul 6, 2022 at 5:55 PM Amit Kapila wrote: > > On Wed, Jul 6, 2022 at 12:19 PM Masahiko Sawada wrote: > > > > On Tue, Jul 5, 2022 at 8:00 PM Amit Kapila wrote: > > > > > > 2. Are we anytime removing transa

Re: Issue with pg_stat_subscription_stats

2022-07-06 Thread Masahiko Sawada
On Thu, Jul 7, 2022 at 1:28 AM Andres Freund wrote: > > On 2022-07-05 14:52:45 -0700, Andres Freund wrote: > > On 2022-07-04 11:01:01 +0900, Masahiko Sawada wrote: > > > I've attached the patch, fix_drop_subscriptions_stats.patch, to fix it. > > > > LGTM. Unless s

Re: Issue with pg_stat_subscription_stats

2022-07-06 Thread Masahiko Sawada
On Thu, Jul 7, 2022 at 12:53 AM Andres Freund wrote: > > Hi, > > On 2022-07-06 11:41:46 +0900, Masahiko Sawada wrote: > > diff --git a/src/test/regress/sql/subscription.sql > > b/src/test/regress/sql/subscription.sql > > index 74c38ead5d..6a46956f6e 100644

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-06 Thread Masahiko Sawada
On Tue, Jul 5, 2022 at 5:09 PM Andres Freund wrote: > > Hi, > > On 2022-07-05 16:33:17 +0900, Masahiko Sawada wrote: > > On Tue, Jul 5, 2022 at 6:18 AM Andres Freund wrote: > > A datum value is convenient to represent both a pointer and a value so > > I used i

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-06 Thread Masahiko Sawada
On Tue, Jul 5, 2022 at 8:00 PM Amit Kapila wrote: > > On Mon, Jul 4, 2022 at 6:12 PM Amit Kapila wrote: > > > > On Mon, May 30, 2022 at 11:13 AM Masahiko Sawada > > wrote: > > > > > > I've attached three POC patches: > > > > > > &g

Re: Issue with pg_stat_subscription_stats

2022-07-05 Thread Masahiko Sawada
On Wed, Jul 6, 2022 at 10:48 AM Andres Freund wrote: > > On 2022-07-06 10:25:02 +0900, Masahiko Sawada wrote: > > > I think most of this could just be pgstat_reset_entry(). > > > > I think pgstat_reset_entry() doesn't work for this case as it skips > > resett

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-05 Thread Masahiko Sawada
On Mon, Jul 4, 2022 at 9:42 PM Amit Kapila wrote: > > On Mon, May 30, 2022 at 11:13 AM Masahiko Sawada > wrote: > > > > I've attached three POC patches: > > > > I think it will be a good idea if you can add a short commit message > at least to say which patc

Re: Issue with pg_stat_subscription_stats

2022-07-05 Thread Masahiko Sawada
On Wed, Jul 6, 2022 at 6:52 AM Andres Freund wrote: > > Hi, > > On 2022-07-04 11:01:01 +0900, Masahiko Sawada wrote: > > I've attached the patch, fix_drop_subscriptions_stats.patch, to fix it. > > LGTM. Unless somebody sees a reason not to, I'm planning to comm

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-05 Thread Masahiko Sawada
On Tue, Jul 5, 2022 at 7:00 AM Andres Freund wrote: > > Hi, > > On 2022-06-28 15:24:11 +0900, Masahiko Sawada wrote: > > In both test cases, There is not much difference between using AVX2 > > and SSE2. The more mode types, the more time it takes for loading the > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-05 Thread Masahiko Sawada
On Tue, Jul 5, 2022 at 6:18 AM Andres Freund wrote: > > Hi, > > On 2022-06-16 13:56:55 +0900, Masahiko Sawada wrote: > > diff --git a/src/backend/lib/radixtree.c b/src/backend/lib/radixtree.c > > new file mode 100644 > > index 00..bf87f932fd > > ---

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-04 Thread Masahiko Sawada
On Mon, Jul 4, 2022 at 2:07 PM Masahiko Sawada wrote: > > On Tue, Jun 28, 2022 at 10:10 PM John Naylor > wrote: > > > > On Tue, Jun 28, 2022 at 1:24 PM Masahiko Sawada > > wrote: > > > > > > > I > > > > suspect other optimizations wou

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-07-03 Thread Masahiko Sawada
On Tue, Jun 28, 2022 at 10:10 PM John Naylor wrote: > > On Tue, Jun 28, 2022 at 1:24 PM Masahiko Sawada wrote: > > > > > I > > > suspect other optimizations would be worth a lot more than using AVX2: > > > - collapsing inner nodes > > > - taking c

Re: Issue with pg_stat_subscription_stats

2022-07-03 Thread Masahiko Sawada
On Sat, Jul 2, 2022 at 9:52 AM Masahiko Sawada wrote: > > > > On Sat, Jul 2, 2022 at 2:53 Andres Freund wrote: >> >> Hi, >> >> On 2022-07-01 16:08:48 +0900, Masahiko Sawada wrote: >> > Yes, my point is that it may be misleading that the subscripti

Re: Issue with pg_stat_subscription_stats

2022-07-01 Thread Masahiko Sawada
On Sat, Jul 2, 2022 at 2:53 Andres Freund wrote: > Hi, > > On 2022-07-01 16:08:48 +0900, Masahiko Sawada wrote: > > Yes, my point is that it may be misleading that the subscription stats > > are created when a subscription is created. > > I think it's important t

Re: Issue with pg_stat_subscription_stats

2022-07-01 Thread Masahiko Sawada
On Fri, Jul 1, 2022 at 3:01 PM Amit Kapila wrote: > > On Fri, Jul 1, 2022 at 7:12 AM Masahiko Sawada wrote: > > > > On Wed, Mar 16, 2022 at 11:34 PM Masahiko Sawada > > wrote: > > > > > > > While looking at this issue again, I realized there seems t

Re: Backup command and functions can cause assertion failure and segmentation fault

2022-07-01 Thread Masahiko Sawada
e to disallow that in the attached patch. +1 @@ -233,6 +233,12 @@ perform_base_backup(basebackup_options *opt, bbsink *sink) StringInfo labelfile; StringInfo tblspc_map_file; backup_manifest_info manifest; + SessionBackupState status = get_backup_status(); + + if (status == SESSION_BACKUP_RUNNING) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), +errmsg("a backup is already in progress in this session"))); I think we can move it to the beginning of SendBaseBackup() so we can avoid bbsink initialization and cleanup in the error case. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Issue with pg_stat_subscription_stats

2022-06-30 Thread Masahiko Sawada
Hi, On Wed, Mar 16, 2022 at 11:34 PM Masahiko Sawada wrote: > > On Wed, Mar 16, 2022 at 8:51 PM Amit Kapila wrote: > > > > On Tue, Mar 15, 2022 at 10:09 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Mar 15, 2022 at 3:34 AM Melanie Plageman &g

Re: Testing autovacuum wraparound (including failsafe)

2022-06-29 Thread Masahiko Sawada
Hi, On Tue, Feb 1, 2022 at 11:58 AM Masahiko Sawada wrote: > > On Fri, Jun 11, 2021 at 10:19 AM Andres Freund wrote: > > > > Hi, > > > > On 2021-06-10 16:42:01 +0300, Anastasia Lubennikova wrote: > > > Cool. Thank you for working on that! > > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-06-28 Thread Masahiko Sawada
Hi, On Mon, Jun 27, 2022 at 8:12 PM John Naylor wrote: > > On Mon, Jun 20, 2022 at 7:57 AM Masahiko Sawada wrote: > > [v3 patch] > > Hi Masahiko, > > Since there are new files, and they are pretty large, I've attached > most specific review comments and ques

Re: Postgres do not allow to create many tables with more than 63-symbols prefix

2022-06-26 Thread Masahiko Sawada
On Fri, Jun 24, 2022 at 2:12 PM Andrey Lepikhov wrote: > > Moved from the pgsql-bugs mailing list [1]. > > On 6/23/22 07:03, Masahiko Sawada wrote: > > Hi, > > > > On Sat, Jun 4, 2022 at 4:03 AM Andrey Lepikhov > > wrote: > >> > >&g

Re: Support logical replication of DDLs

2022-06-23 Thread Masahiko Sawada
On Thu, Jun 23, 2022 at 7:00 PM Amit Kapila wrote: > > On Wed, Jun 22, 2022 at 11:09 AM Masahiko Sawada > wrote: > > > > I've attached a WIP patch for adding regression tests for DDL deparse. > > The patch can be applied on > > v9-0001-Functions-to-depars

Re: Fix instability in subscription regression test

2022-06-23 Thread Masahiko Sawada
tioned" has finished > > From the Log, I can see it query the target table before the table sync is > over. So, I think the reason is that we didn't wait for table sync to > finish after refreshing the publication. Sorry for not catching that > ealier. Here is a patch to fix it. +1 The patch looks good to me. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: pg_page_repair: a tool/extension to repair corrupted pages in postgres with streaming/physical replication

2022-06-22 Thread Masahiko Sawada
rds from backward but it could take a quite long time. Regards, [1] https://github.com/MasahikoSawada/pgtools/tree/master/page_repair [2] https://www.postgresql.org/message-id/CAOBaU_aVvMjQn%3Dge5qPiJOPMmOj5%3Dii3st5Q0Y%2BWuLML5sR17w%40mail.gmail.com -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Missing reference to pgstat_replslot.c in pgstat.c

2022-06-22 Thread Masahiko Sawada
On Wed, Jun 22, 2022 at 3:29 PM Drouvot, Bertrand wrote: > > Hi hackers, > > I think there's a missing reference to pgstat_replslot.c in pgstat.c. > > Attached a tiny patch to fix it. +1 Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Support logical replication of DDLs

2022-06-21 Thread Masahiko Sawada
the regression tests, load the deparsed DDLs to another database cluster, dump both databases, and compare the dumps. Since the patch doesn't support deparsing all DDLs and there is a bug[2], the attached regression test does CREATE TABLE and some ALTER TABLE instead of running regression tests. Regard

Re: Add index scan progress to pg_stat_progress_vacuum

2022-06-20 Thread Masahiko Sawada
On Mon, Jun 6, 2022 at 11:42 PM Robert Haas wrote: > > On Thu, May 26, 2022 at 11:43 AM Masahiko Sawada > wrote: > > Another idea I came up with is that we can wait for all index vacuums > > to finish while checking and updating the progress informatio

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-06-19 Thread Masahiko Sawada
Hi, On Thu, Jun 16, 2022 at 4:30 PM John Naylor wrote: > > On Thu, Jun 16, 2022 at 11:57 AM Masahiko Sawada > wrote: > > I've attached an updated version patch that changes the configure > > script. I'm still studying how to support AVX2 on msvc build. Also, > >

Re: Skipping logical replication transactions on subscriber side

2022-06-16 Thread Masahiko Sawada
On Thu, Jun 16, 2022 at 2:27 AM Robert Haas wrote: > > On Tue, Jun 14, 2022 at 3:54 AM Masahiko Sawada wrote: > > > AFAICS, we could do that by: > > > > > > 1. De-supporting platforms that have this problem, or > > > 2. Introducing new typalig

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-06-15 Thread Masahiko Sawada
On Wed, May 25, 2022 at 11:48 AM Masahiko Sawada wrote: > > On Tue, May 10, 2022 at 6:58 PM John Naylor > wrote: > > > > On Tue, May 10, 2022 at 8:52 AM Masahiko Sawada > > wrote: > > > > > > Overall, radix tree implementations have good numb

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-06-14 Thread Masahiko Sawada
On Tue, Jun 14, 2022 at 3:57 PM Amit Kapila wrote: > > On Mon, Jun 13, 2022 at 8:29 AM Masahiko Sawada wrote: > > > > On Tue, Jun 7, 2022 at 9:32 PM Amit Kapila wrote: > > > > > > On Mon, May 30, 2022 at 11:13 AM Masahiko Sawada > > > wrote: >

Re: Skipping logical replication transactions on subscriber side

2022-06-14 Thread Masahiko Sawada
n all platforms Introducing new typalign values seems a good idea to me as it's more future-proof. Will this item be for PG16, right? The main concern seems that what this test case enforces would be nuisance when introducing a new system catalog or a new column to the existing catalog but given we're in post PG15-beta1 it is unlikely to happen in PG15. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-06-12 Thread Masahiko Sawada
On Tue, Jun 7, 2022 at 9:32 PM Amit Kapila wrote: > > On Mon, May 30, 2022 at 11:13 AM Masahiko Sawada > wrote: > > > > On Wed, May 25, 2022 at 12:11 PM Masahiko Sawada > > wrote: > > > > > > > poc_add_regression_tests.patch adds regress

Re: Add index scan progress to pg_stat_progress_vacuum

2022-06-02 Thread Masahiko Sawada
ery page and used not only by vacuum such as analyze, but it seems to be worth trying. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-31 Thread Masahiko Sawada
per subscription but it is quite possible that I am > missing something and there are problems due to which we can't use > that approach. I prefer the first idea as it's simpler than the second one. I don't see any concurrency problem so far unless I'm not missing something. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Support logical replication of DDLs

2022-05-30 Thread Masahiko Sawada
I think it's OK even if it happens, > > we'll find out > > why and fix it. > > > > FWIW, both these cases could be handled with the deparsing approach, > and the handling related to the drop of multiple tables where only a > few are published is already done in the last POC patch shared by Ajin > [1]. > Right. So I'm inclined to think that deparsing approach is better from this point as well as the point mentioned by Álvaro before[1]. Regards, [1] https://www.postgresql.org/message-id/202204081134.6tcmf5cxl3sz%40alvherre.pgsql -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-05-29 Thread Masahiko Sawada
On Wed, May 25, 2022 at 12:11 PM Masahiko Sawada wrote: > > On Tue, May 24, 2022 at 2:18 PM Amit Kapila wrote: > > > > On Tue, May 24, 2022 at 7:58 AM Masahiko Sawada > > wrote: > > > > > > On Mon, May 23, 2022 at 2:39 PM Amit Kapila > > > w

Re: Support logical replication of DDLs

2022-05-27 Thread Masahiko Sawada
is should be rare and I think it's OK even if it happens, > we'll find out > why and fix it. I'm not sure it's rare since replicating a subset of tables is a common use case of logical replication. But even if we want to go this way I think we should consider how to fix it at this stage, otherwise we will end up redesigning it. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Add index scan progress to pg_stat_progress_vacuum

2022-05-26 Thread Masahiko Sawada
eaders backend only. I was concerned that the leader process could report the wrong progress if updating and checking index status happen concurrently. But I think it should be fine since we can read PVIndVacStatus atomically. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Support logical replication of DDLs

2022-05-26 Thread Masahiko Sawada
ior. This seems to require a different solution from what the patch does for the problem you mentioned such as "DDL involving multiple tables where only some tables are replicated”. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-05-24 Thread Masahiko Sawada
On Tue, May 24, 2022 at 2:18 PM Amit Kapila wrote: > > On Tue, May 24, 2022 at 7:58 AM Masahiko Sawada wrote: > > > > On Mon, May 23, 2022 at 2:39 PM Amit Kapila wrote: > > > > > > On Mon, May 23, 2022 at 10:03 AM Kyotaro Horiguchi > > > wrote:

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-05-24 Thread Masahiko Sawada
On Tue, May 10, 2022 at 6:58 PM John Naylor wrote: > > On Tue, May 10, 2022 at 8:52 AM Masahiko Sawada wrote: > > > > Overall, radix tree implementations have good numbers. Once we got an > > agreement on moving in this direction, I'll start a new thread for > > th

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-05-23 Thread Masahiko Sawada
R6KYvkf663kg6EqW-f6sz1w%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAD21AoD00wV4gt-53ze%2BZB8n4bqJrdH8J_UnDHddy8S2A%2Ba25g%40mail.gmail.com [3] https://www.postgresql.org/message-id/20211008.165055.1621145185927268721.horikyota.ntt%40gmail.com -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Intermittent buildfarm failures on wrasse

2022-05-18 Thread Masahiko Sawada
On Sun, May 15, 2022 at 12:29 AM Alvaro Herrera wrote: > > On 2022-Apr-20, Masahiko Sawada wrote: > > > > MyProc->statusFlags = (MyProc->statusFlags & ~PROC_XMIN_FLAGS) | > > > (proc->statusFlags & PROC_XMIN_FLAGS); > &g

Re: Support logical replication of DDLs

2022-05-11 Thread Masahiko Sawada
On Wed, May 11, 2022 at 6:15 PM Amit Kapila wrote: > > On Wed, May 11, 2022 at 1:09 PM Masahiko Sawada wrote: > > > > On Mon, May 9, 2022 at 6:05 PM Alvaro Herrera > > wrote: > > > > > > On 2022-May-08, Dilip Kumar wrote: > > > > >

Re: Support logical replication of DDLs

2022-05-11 Thread Masahiko Sawada
mplement such regression tests in order to check not only the existing commands but also future changes. I'll try it if no one is working on it, and let us see if there are challenges. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Support logical replication of DDLs

2022-05-10 Thread Masahiko Sawada
op the deparsing utility as an extension itself. IIUC the event trigger can already provide such flexibility. That is, one could create an extension that creates an event trigger and in the event trigger function it deparses the SQL parse tree to a SQL command that can be run in MySQL. While having such flexibility, I’m fine with having built-in deparsing utility in the core. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-10 Thread Masahiko Sawada
On Tue, May 10, 2022 at 5:59 PM Amit Kapila wrote: > > On Tue, May 10, 2022 at 10:39 AM Masahiko Sawada > wrote: > > > > On Wed, May 4, 2022 at 8:44 AM Peter Smith wrote: > > > > > > On Tue, May 3, 2022 at 5:16 PM Peter Smith wrote: > > &g

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-10 Thread Masahiko Sawada
On Tue, May 10, 2022 at 6:10 PM Amit Kapila wrote: > > On Tue, May 10, 2022 at 10:35 AM Masahiko Sawada > wrote: > > > > On Wed, May 4, 2022 at 12:50 PM Amit Kapila wrote: > > > > > > On Tue, May 3, 2022 at 9:45 AM Amit Kapila > > > wrote

Re: Support logical replication of DDLs

2022-05-10 Thread Masahiko Sawada
he table schema somehow, then starts the initial data copy. After that, syncworker or applyworker applies DML/DDL changes while catching up and streaming changes, respectively. Probably we can have it optional whether to copy schema only, data only, or both. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-09 Thread Masahiko Sawada
nfigure how many apply bgworkers can be used per subscription or in the whole system? Like max_sync_workers_per_subscription, is it better to have a configuration parameter or a subscription option for that? If so, setting it to 0 probably means to disable the parallel apply feature. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-09 Thread Masahiko Sawada
On Wed, May 4, 2022 at 12:50 PM Amit Kapila wrote: > > On Tue, May 3, 2022 at 9:45 AM Amit Kapila wrote: > > > > On Mon, May 2, 2022 at 5:06 PM Masahiko Sawada > > wrote: > > > > > > On Mon, May 2, 2022 at 6:09 PM Amit Kapila > > > wro

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-05-09 Thread Masahiko Sawada
Hi, On Sun, Feb 13, 2022 at 12:39 PM Andres Freund wrote: > > On 2022-02-13 12:36:13 +0900, Masahiko Sawada wrote: > > Actually, I'm working on simplifying and improving radix tree > > implementation for PG16 dev cycle. From the discussion so far I think > > it's better

Re: Logical replication timeout problem

2022-05-09 Thread Masahiko Sawada
On Mon, May 9, 2022 at 3:47 PM Amit Kapila wrote: > > On Fri, May 6, 2022 at 12:42 PM wangw.f...@fujitsu.com > wrote: > > > > On Fri, May 6, 2022 at 9:54 AM Masahiko Sawada > > wrote: > > > On Wed, May 4, 2022 at 7:18 PM Amit Kapila > > > wrote: &

Re: Logical replication timeout problem

2022-05-05 Thread Masahiko Sawada
On Wed, May 4, 2022 at 7:18 PM Amit Kapila wrote: > > On Mon, May 2, 2022 at 8:07 AM Masahiko Sawada wrote: > > > > On Mon, May 2, 2022 at 11:32 AM Amit Kapila wrote: > > > > > > > > > So, shall we go back to the previous approach of using a separa

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-02 Thread Masahiko Sawada
On Mon, May 2, 2022 at 6:09 PM Amit Kapila wrote: > > On Mon, May 2, 2022 at 11:47 AM Masahiko Sawada wrote: > > > > On Fri, Apr 8, 2022 at 6:14 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Wednesday, April 6, 2022 1:20 PM Amit Kapila > &g

Re: Perform streaming logical transactions by background workers and parallel apply

2022-05-02 Thread Masahiko Sawada
* handle the case when there is no more worker available > (might need spill the data to the temp file in this case) > * some potential bugs Are you planning to support "Transaction dependency" Amit mentioned in his first mail in this patch? IIUC since the background apply wo

Re: Add index scan progress to pg_stat_progress_vacuum

2022-05-01 Thread Masahiko Sawada
PARALLEL_INDVAC_STATUS_COMPLETED. BTW, currently we don't need a lock for touching index status since each worker touches different indexes. But after this patch, the leader will touch all index status, do we need a lock for that? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Logical replication timeout problem

2022-05-01 Thread Masahiko Sawada
On Mon, May 2, 2022 at 11:32 AM Amit Kapila wrote: > > On Mon, May 2, 2022 at 7:33 AM Masahiko Sawada wrote: > > On Thu, Apr 28, 2022 at 7:01 PM houzj.f...@fujitsu.com > > wrote: > > > > > > Hi Sawada-san, Wang > > > > > > I was looking

Re: Logical replication timeout problem

2022-05-01 Thread Masahiko Sawada
On Thu, Apr 28, 2022 at 7:01 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, April 20, 2022 3:21 PM Masahiko Sawada > wrote: > > > > BTW the changes in > > REL_14_v1-0001-Fix-the-logical-replication-timeout-during-large-.patch, > > adding end_xact to Logical

Re: Logical replication timeout problem

2022-04-20 Thread Masahiko Sawada
On Thu, Apr 21, 2022 at 11:19 AM Amit Kapila wrote: > > On Wed, Apr 20, 2022 at 6:22 PM Masahiko Sawada wrote: > > > > On Wed, Apr 20, 2022 at 7:12 PM Amit Kapila wrote: > > > > > > > > I think it would > > > be then better to have it in the

Re: Logical replication timeout problem

2022-04-20 Thread Masahiko Sawada
On Wed, Apr 20, 2022 at 7:12 PM Amit Kapila wrote: > > On Wed, Apr 20, 2022 at 2:38 PM Amit Kapila wrote: > > > > On Wed, Apr 20, 2022 at 12:51 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Apr 20, 2022 at 11:46 AM wangw.f...@fujitsu.com > >

Re: Logical replication timeout problem

2022-04-20 Thread Masahiko Sawada
ter to put it after fast_forward where the new field should fall within the padding space. BTW the changes in REL_14_v1-0001-Fix-the-logical-replication-timeout-during-large-.patch, adding end_xact to LogicalDecodingContext, seems good to me and it might be better than the approach of v17 patch from plugin developers’ perspective? This is because they won’t need to pass true/false to end_xact of OutputPluginUpdateProgress(). Furthermore, if we do what we do in update_replication_progress() in OutputPluginUpdateProgress(), what plugins need to do will be just to call OutputPluginUpdate() in every callback and they don't need to have the CHANGES_THRESHOLD logic. What do you think? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Intermittent buildfarm failures on wrasse

2022-04-19 Thread Masahiko Sawada
; > Perhaps the latter is more future-proof. Copying only xmin-related flags in this way also makes sense to me and there is no problem at least for now. A note would be that when we introduce a new flag that needs to be copied in the future, we need to make sure to add it to PROC_XMIN_FLAGS so it is copied. Otherwise a similar issue we fixed by 0f0cfb494004befb0f6e could happen again. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Logical replication timeout problem

2022-04-18 Thread Masahiko Sawada
On Mon, Apr 18, 2022 at 3:16 PM wangw.f...@fujitsu.com wrote: > > On Mon, Apr 18, 2022 at 00:35 PM Masahiko Sawada > wrote: > > On Mon, Apr 18, 2022 at 1:01 PM Amit Kapila wrote: > > > > > > On Thu, Apr 14, 2022 at 5:50 PM Masahiko Sawada > > wrote: &g

Re: Column Filtering in Logical Replication

2022-04-18 Thread Masahiko Sawada
On Mon, Apr 18, 2022 at 8:04 PM Amit Kapila wrote: > > On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila wrote: > > > > On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada > > wrote: > > > > > > > > The other part of the puzzle is the below check in the c

Re: Logical replication timeout problem

2022-04-17 Thread Masahiko Sawada
On Mon, Apr 18, 2022 at 1:01 PM Amit Kapila wrote: > > On Thu, Apr 14, 2022 at 5:50 PM Masahiko Sawada wrote: > > > > On Wed, Apr 13, 2022 at 7:45 PM Amit Kapila wrote: > > > > > > On Mon, Apr 11, 2022 at 12:09 PM wangw.f...@fujitsu.com > > >

<    3   4   5   6   7   8   9   10   11   12   >