Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Masahiko Sawada
On Fri, Nov 18, 2022 at 1:47 PM Amit Kapila wrote: > > On Fri, Nov 18, 2022 at 8:01 AM Peter Smith wrote: > > > > On Fri, Nov 18, 2022 at 11:36 AM Masahiko Sawada > > wrote: > > > > > ... > > > --- > > > The streaming parameter

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-17 Thread Masahiko Sawada
ique constraint "test1_c_idx" 2022-11-17 17:30:23.490 JST [3814552] DETAIL: Key (c)=(1) already exists. 2022-11-17 17:30:23.490 JST [3814552] CONTEXT: processing remote data for replication origin "pg_16390" during message type "INSERT" for replication target relatio n "public.test1" in transaction 731 2022-11-17 17:30:23.490 JST [3814550] ERROR: duplicate key value violates unique constraint "test1_c_idx" 2022-11-17 17:30:23.490 JST [3814550] DETAIL: Key (c)=(1) already exists. 2022-11-17 17:30:23.490 JST [3814550] CONTEXT: processing remote data for replication origin "pg_16390" during message type "INSERT" for replication target relatio n "public.test1" in transaction 731 parallel apply worker Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-16 Thread Masahiko Sawada
On Wed, Nov 16, 2022 at 4:39 PM John Naylor wrote: > > > On Wed, Nov 16, 2022 at 12:33 PM Masahiko Sawada > wrote: > > > > On Wed, Nov 16, 2022 at 1:46 PM John Naylor > > wrote: > > > > > > > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread Masahiko Sawada
On Wed, Nov 16, 2022 at 2:17 PM John Naylor wrote: > > > > On Wed, Nov 16, 2022 at 11:46 AM John Naylor > wrote: > > > > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada > > wrote: > > > Thanks! Please let me know if there is somethin

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-15 Thread Masahiko Sawada
On Wed, Nov 16, 2022 at 1:46 PM John Naylor wrote: > > > On Tue, Nov 15, 2022 at 11:59 AM Masahiko Sawada > wrote: > > Thanks! Please let me know if there is something I can help with. > > I didn't get very far because the tests fail on 0004 in rt_verify_node: >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-14 Thread Masahiko Sawada
On Mon, Nov 14, 2022 at 10:00 PM John Naylor wrote: > > On Mon, Nov 14, 2022 at 3:44 PM Masahiko Sawada wrote: > > > > 0004 patch is a new patch supporting a pointer tagging of the node > > kind. Also, it introduces rt_node_ptr we discussed so that internal > >

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-08 Thread Masahiko Sawada
On Sat, Nov 5, 2022 at 6:23 PM John Naylor wrote: > > On Fri, Nov 4, 2022 at 10:25 PM Masahiko Sawada wrote: > > > > For parallel heap pruning, multiple workers will insert key-value > > pairs to the radix tree concurrently. The simplest solution would be a > >

Re: Add index scan progress to pg_stat_progress_vacuum

2022-11-07 Thread Masahiko Sawada
rg); +void *parallel_progress_arg; } ParallelContext; With the above change I suggested, I think we won't need to have a callback function in ParallelContext. Instead, I think we can have index-AMs call parallel_vacuum_report() if report_parallel_progress is true. Regards, -- Masahiko Sawada Amazon W

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-07 Thread Masahiko Sawada
On Thu, Nov 3, 2022 at 10:06 PM houzj.f...@fujitsu.com wrote: > > On Wednesday, November 2, 2022 10:50 AM Masahiko Sawada > wrote: > > > > On Mon, Oct 24, 2022 at 8:42 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Oct 12, 2022 at 3:04 PM Amit K

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-06 Thread Masahiko Sawada
On Mon, Nov 7, 2022 at 12:58 PM Amit Kapila wrote: > > On Mon, Nov 7, 2022 at 8:26 AM Masahiko Sawada wrote: > > > > On Sun, Nov 6, 2022 at 3:40 PM houzj.f...@fujitsu.com > > wrote: > > > > > > On Saturday, November 5, 2022 1:43 PM Amit Kapila > >

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-06 Thread Masahiko Sawada
e need it for our purpose, won't subscription id and > > xid can uniquely identify the tag? > > I agree that it could be better to have a new lock tag. Another point is that > the remote xid and Local xid could be the same in some rare cases, so I think > we might need to add another identifier to make it unique. > > Maybe : > locktag_field1 : subscription oid > locktag_field2 : xid(remote or local) > locktag_field3 : 0(lock for stream block)/1(lock for transaction) Or I think we can use locktag_field2 for remote xid and locktag_field3 for local xid. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-04 Thread Masahiko Sawada
On Thu, Nov 3, 2022 at 1:59 PM John Naylor wrote: > > On Mon, Oct 31, 2022 at 12:47 PM Masahiko Sawada > wrote: > > > > I've attached v8 patches. 0001, 0002, and 0003 patches incorporated > > the comments I got so far. 0004 patch is a DSA support patch for PoC. >

Re: Fix comments atop ReorderBufferAddInvalidations

2022-11-03 Thread Masahiko Sawada
served it while working on a fix for commit > 16b1fe0037. Thank you for the patch. It looks good to me. I think we can backpatch it to avoid confusion in future. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-01 Thread Masahiko Sawada
On Mon, Oct 24, 2022 at 8:42 PM Masahiko Sawada wrote: > > On Wed, Oct 12, 2022 at 3:04 PM Amit Kapila wrote: > > > > On Tue, Oct 11, 2022 at 5:52 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Oct 7, 2022 at 2:00 PM Amit Kapila > > >

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-27 Thread Masahiko Sawada
On Thu, Oct 27, 2022 at 11:34 AM shiy.f...@fujitsu.com wrote: > > On Wed, Oct 26, 2022 7:19 PM Amit Kapila wrote: > > > > On Tue, Oct 25, 2022 at 8:38 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Oct 21, 2022 at 6:32 PM houzj.f...@fujitsu.

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-26 Thread Masahiko Sawada
On Wed, Oct 26, 2022 at 8:06 PM John Naylor wrote: > > On Mon, Oct 24, 2022 at 12:54 PM Masahiko Sawada > wrote: > > > I've attached updated PoC patches for discussion and cfbot. From the > > previous version, I mainly changed the following things: > &g

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-24 Thread Masahiko Sawada
ee (leader) apply workers received START_STREAM and launched their parallel apply workers. However, two of them finished without applying any data. I think this behaviour looks problematic since it wastes workers and rather decreases the apply performance if the changes are not large. Is it worth conside

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-24 Thread Masahiko Sawada
On Wed, Oct 12, 2022 at 3:04 PM Amit Kapila wrote: > > On Tue, Oct 11, 2022 at 5:52 AM Masahiko Sawada wrote: > > > > On Fri, Oct 7, 2022 at 2:00 PM Amit Kapila wrote: > > > > > > About your point that having different partition structures for > > &g

Re: START_REPLICATION SLOT causing a crash in an assert build

2022-10-21 Thread Masahiko Sawada
On Thu, Oct 20, 2022 at 6:54 AM Andres Freund wrote: > > Hi, > > On 2022-10-13 15:57:28 +0900, Masahiko Sawada wrote: > > I've attached an updated patch. I've added the common function to > > start pg_recvlogical and wait for it to become active. Please revie

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-20 Thread Masahiko Sawada
On Thu, Oct 20, 2022 at 6:57 PM Amit Kapila wrote: > > On Wed, Oct 19, 2022 at 9:40 AM Masahiko Sawada wrote: > > > > I've attached patches for Change-3 with a test case. Please review them as > > well. > > > > The patch looks mostly good to me

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-20 Thread Masahiko Sawada
On Thu, Oct 20, 2022 at 8:09 PM Amit Kapila wrote: > > On Wed, Oct 19, 2022 at 4:47 PM Amit Kapila wrote: > > > > On Wed, Oct 19, 2022 at 1:08 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Oct 19, 2022 at 11:58 AM Masahiko Sawada > > &g

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-19 Thread Masahiko Sawada
On Wed, Oct 19, 2022 at 11:58 AM Masahiko Sawada wrote: > > On Tue, Oct 18, 2022 at 9:53 PM Masahiko Sawada wrote: > > > > On Tue, Oct 18, 2022 at 7:49 PM Amit Kapila wrote: > > > > > > On Tue, Oct 18, 2022 at 1:45 PM Masahiko Sawada > > > wrote

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-18 Thread Masahiko Sawada
On Wed, Oct 19, 2022 at 11:58 AM Masahiko Sawada wrote: > > On Tue, Oct 18, 2022 at 9:53 PM Masahiko Sawada wrote: > > > > On Tue, Oct 18, 2022 at 7:49 PM Amit Kapila wrote: > > > > > > On Tue, Oct 18, 2022 at 1:45 PM Masahiko Sawada > > > wrote

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-18 Thread Masahiko Sawada
On Tue, Oct 18, 2022 at 9:53 PM Masahiko Sawada wrote: > > On Tue, Oct 18, 2022 at 7:49 PM Amit Kapila wrote: > > > > On Tue, Oct 18, 2022 at 1:45 PM Masahiko Sawada > > wrote: > > > > > > > > > > > I think because the test case propose

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-18 Thread Masahiko Sawada
On Tue, Oct 18, 2022 at 7:56 PM Amit Kapila wrote: > > On Mon, Oct 17, 2022 at 7:05 AM Masahiko Sawada wrote: > > > > On Thu, Oct 13, 2022 at 4:08 PM Amit Kapila wrote: > > > > > > --- a/src/backend/replication/logical/decode.c > > > +

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-18 Thread Masahiko Sawada
On Tue, Oct 18, 2022 at 7:49 PM Amit Kapila wrote: > > On Tue, Oct 18, 2022 at 1:45 PM Masahiko Sawada wrote: > > > > > > > > I think because the test case proposed needs all three changes, we can > > > push the change-1 without a test case and then as a se

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-18 Thread Masahiko Sawada
On Tue, Oct 18, 2022 at 1:07 PM Amit Kapila wrote: > > On Tue, Oct 18, 2022 at 6:29 AM Masahiko Sawada wrote: > > > > On Mon, Oct 17, 2022 at 4:40 PM Amit Kapila wrote: > > > > > > > > > IIUC, here you are speaking of three different changes. Cha

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-17 Thread Masahiko Sawada
On Mon, Oct 17, 2022 at 4:40 PM Amit Kapila wrote: > > On Wed, Oct 12, 2022 at 11:18 AM Masahiko Sawada > wrote: > > > > Please note that to pass the new regression tests, the fix proposed in > > a related thread[1] is required. Particularly, we need:

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-16 Thread Masahiko Sawada
On Thu, Oct 13, 2022 at 4:08 PM Amit Kapila wrote: > > On Wed, Oct 12, 2022 at 11:18 AM Masahiko Sawada > wrote: > > > > Summarizing this issue, the assertion check in AssertTXNLsnOrder() > > fails as reported because the current logical decoding cannot properl

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-14 Thread Masahiko Sawada
. > > On Fri, Oct 7, 2022 at 3:09 PM Masahiko Sawada wrote: > > [fixed benchmarks] > > Thanks for that! Now I can show clear results on some aspects in a simple > way. The attached patches (apply on top of v6) are not intended to be > incorporated as-is quite yet,

Re: START_REPLICATION SLOT causing a crash in an assert build

2022-10-13 Thread Masahiko Sawada
On Thu, Oct 13, 2022 at 1:21 AM Andres Freund wrote: > > Hi, > > On 2022-10-11 17:10:52 +0900, Masahiko Sawada wrote: > > +# Reset the replication slot statistics. > > +$node->safe_psql('postgres', > > + "SELECT pg_stat_reset_replication_slot('regress

Re: test_decoding assertion failure for the loss of top-sub transaction relationship

2022-10-12 Thread Masahiko Sawada
confusing you. I meant that even if we agreed with the patch I proposed there, your patch is still required to fix the issue. Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Add index scan progress to pg_stat_progress_vacuum

2022-10-12 Thread Masahiko Sawada
patches. One for adding the new statistics of the number of vacuumed indexes to pg_stat_progress_vacuum and another one for adding new statistics view pg_stat_progress_vacuum_index. Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: test_decoding assertion failure for the loss of top-sub transaction relationship

2022-10-12 Thread Masahiko Sawada
osed fix, the patch proposed by Kuroda-san is required for HEAD, REL14, and REL15, in order to fix the assertion failure in SnapBuildCommitTxn(). Regards, [1] https://www.postgresql.org/message-id/CAD21AoA1gV9pfu8hoXpTQBWH8uEMRg_F_MKM%2BU3Sr0HnyH4AUQ%40mail.gmail.com -- Masahiko Sawada

Re: TRAP: FailedAssertion("prev_first_lsn < cur_txn->first_lsn", File: "reorderbuffer.c", Line: 927, PID: 568639)

2022-10-11 Thread Masahiko Sawada
s that we think are top-level. And then we > > get the commit for the actual top-level transaction. AFAICS that won't > > clean-up the subxact, and we end up with a lingering TXN. > > > > I think we will clean up such a subxact. Such a xact should be skipped > via DecodeTXNN

Re: test_decoding assertion failure for the loss of top-sub transaction relationship

2022-10-11 Thread Masahiko Sawada
etween top-level transaction and subtransaction yet. When decoding the # commit record of the top-level transaction, we must force the top-level transaction # to do timetravel since one of its subtransactions is marked as containing catalog changes. --- + elog(DEBUG2, "forced transaction %u to do timetravel due to one of its subtransaction", + xid); + needs_timetravel = true; I think "one of its subtransaction" should be "one of its subtransactions". Regards, -- Masahiko Sawada

Re: START_REPLICATION SLOT causing a crash in an assert build

2022-10-11 Thread Masahiko Sawada
have been removed > > > I wonder how much it'd take to teach isolationtester to handle the replication > protocol... I think we can do these tests by using pg_recvlogical in TAP tests. I've attached a patch to do that. Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Sour

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-10 Thread Masahiko Sawada
On Fri, Oct 7, 2022 at 2:00 PM Amit Kapila wrote: > > On Fri, Oct 7, 2022 at 8:47 AM Masahiko Sawada wrote: > > > > On Thu, Oct 6, 2022 at 9:04 PM houzj.f...@fujitsu.com > > wrote: > > > > > > I think the root reason for this kind of deadlock proble

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-07 Thread Masahiko Sawada
On Fri, Oct 7, 2022 at 2:29 PM John Naylor wrote: > > On Fri, Sep 16, 2022 at 1:01 PM Masahiko Sawada wrote: > > In addition to two patches, I've attached the third patch. It's not > > part of radix tree implementation but introduces a contrib module > > bench_radix_tre

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-06 Thread Masahiko Sawada
On Thu, Oct 6, 2022 at 9:04 PM houzj.f...@fujitsu.com wrote: > > > > > -Original Message- > > From: Masahiko Sawada > > Sent: Thursday, October 6, 2022 4:07 PM > > To: Hou, Zhijie/侯 志杰 > > Cc: Amit Kapila ; Wang, Wei/王 威 > > ; Peter Smith ; D

Re: START_REPLICATION SLOT causing a crash in an assert build

2022-10-06 Thread Masahiko Sawada
ts entry, pgstat_acquire_replslot() is no longer necessary? Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Issue with pg_stat_subscription_stats

2022-10-06 Thread Masahiko Sawada
> Agreed. > > > > > Amit or Andres, are you planning to double-check and perhaps merge > > > this patch to take care of the inconsistency? > > > > I'll run it through CI and then to master unless somebody pipes up in the > > meantime. > > And pushed. Thanks all! Thanks! Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Perform streaming logical transactions by background workers and parallel apply

2022-10-06 Thread Masahiko Sawada
* TX-1 BEGIN; INSERT INTO p SELECT 1 FROM generate_series(1, 1); --- changes are streamed * TX-2 BEGIN; TRUNCATE c2; --- wait for a lock on c2 * TX-1 INSERT INTO p SELECT 1 FROM generate_series(1, 1); COMMIT; This might not be a common case in practice but it could mean that there is

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-06 Thread Masahiko Sawada
On Wed, Oct 5, 2022 at 6:40 PM John Naylor wrote: > > > On Wed, Oct 5, 2022 at 1:46 PM Masahiko Sawada wrote: > > > > On Wed, Sep 28, 2022 at 12:49 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Sep 23, 2022 at 12:11 AM John Naylor > > &

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-05 Thread Masahiko Sawada
On Wed, Sep 28, 2022 at 12:49 PM Masahiko Sawada wrote: > > On Fri, Sep 23, 2022 at 12:11 AM John Naylor > wrote: > > > > > > On Thu, Sep 22, 2022 at 11:46 AM John Naylor > > wrote: > > > One thing I want to try soon is storing fewer than 16/32 etc e

Re: Improve description of XLOG_RUNNING_XACTS

2022-10-03 Thread Masahiko Sawada
> things telling only if the list of overflowed. So let's stick to > that. Why are only subtransaction information in XLOG_RUNNING_XACTS limited? I think we have other information shown without bounds such as lock information in XLOG_STANDBY_LOCK and invalidation messages in XLOG_INVALIDATIONS. R

Re: Small miscellaneous fixes

2022-10-03 Thread Masahiko Sawada
ess var declaration record (src/backend/utils/misc/guc.c) > The var record is never really used. Three changes look good to me. > > 4. Fix declaration volatile signal var (src/bin/pgbench/pgbench.c) > Like how to commit 5ac9e86, this is a similar case. The same is true also for alarm_tri

Re: [PATCH] Log details for client certificate failures

2022-10-02 Thread Masahiko Sawada
On Sat, Oct 1, 2022 at 7:53 PM Peter Eisentraut wrote: > > On 29.09.22 06:52, Masahiko Sawada wrote: > > While this seems a future-proof idea, I wonder if it might be overkill > > since we don't need to worry about accumulation of leaked memory in > > this case. Given tha

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-10-02 Thread Masahiko Sawada
On Mon, Oct 3, 2022 at 2:04 AM Andres Freund wrote: > > Hi, > > On 2022-09-16 15:00:31 +0900, Masahiko Sawada wrote: > > I've updated the radix tree patch. It's now separated into two patches. > > cfbot notices a compiler warning: > https://cirrus-ci.com/task/624790

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-09-30 Thread Masahiko Sawada
On Wed, Sep 28, 2022 at 3:18 PM John Naylor wrote: > > On Wed, Sep 28, 2022 at 10:49 AM Masahiko Sawada > wrote: > > > BTW We need to consider not only aset/slab but also DSA since we > > allocate dead tuple TIDs on DSM in parallel vacuum cases. FYI DSA uses > &

Re: [PATCH] Log details for client certificate failures

2022-09-28 Thread Masahiko Sawada
On Thu, Sep 29, 2022 at 1:43 AM Jacob Champion wrote: > > On Tue, Sep 27, 2022 at 6:14 PM Masahiko Sawada wrote: > > No. Since cluster_name is PGC_POSTMATER, we leak a little postmaster > > memory only once when starting up. application_name is PGC_USERSET but > > si

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-09-27 Thread Masahiko Sawada
560, 1816, 2048, /* 4 classes separated by ~256 bytes */ 2616, 3120, 3640, 4096, /* 4 classes separated by ~512 bytes */ 5456, 6552, 7280, 8192 /* 4 classes separated by ~1024 bytes */ }; node256 will be classed as 2616, which is still not good. Anyway, I'll implement DSA support for radix tree. Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: [PATCH] Log details for client certificate failures

2022-09-27 Thread Masahiko Sawada
On Wed, Sep 28, 2022 at 4:44 AM Jacob Champion wrote: > > On Tue, Sep 27, 2022 at 1:51 AM Masahiko Sawada wrote: > > I think we can fix it by the attached patch but I'd like to discuss > > whether it's worth fixing it. > > Whoops. So every time it's changed, we leak a

Re: [PATCH] Log details for client certificate failures

2022-09-27 Thread Masahiko Sawada
x7A5CF3: check_cluster_name (variable.c:1061) by 0xCAF7CD: call_string_check_hook (guc.c:6365) by 0xCAA724: InitializeOneGUCOption (guc.c:1439) by 0xCAA0ED: InitializeGUCOptions (guc.c:1268) by 0x99B245: PostmasterMain (postmaster.c:691) by 0x858896: main (main.c:197) I think we ca

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-09-22 Thread Masahiko Sawada
le reason for the recent "pg_lfind_ge" function. Good point. While keeping the chunks in the small nodes in sorted order is useful for visiting all keys in sorted order, additional branches and memmove calls could be slow. Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-09-20 Thread Masahiko Sawada
On Fri, Sep 16, 2022 at 4:54 PM John Naylor wrote: > > On Fri, Sep 16, 2022 at 1:01 PM Masahiko Sawada wrote: > > > > On Mon, Aug 15, 2022 at 10:39 PM John Naylor > > wrote: > > > > > > bool, buth = and <=. Should be pretty close. Also, i believe

Fix incorrect variable type for origin_id

2022-09-19 Thread Masahiko Sawada
Hi, I realized that there are some places where we use XLogRecPtr for variables for replication origin id. The attached patch fixes them to use RepOriginiId instead. Regards, -- Masahiko Sawada PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-16 Thread Masahiko Sawada
ted page offset > numbers in a separate area, grouped by their canonical freeze plan > (which is a normalized version of the information currently stored in > xl_heap_freeze_tuple). True. I've not looked at the patch in depth yet but I think we need regression tests for this. Regards,

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-09-16 Thread Masahiko Sawada
On Mon, Aug 15, 2022 at 10:39 PM John Naylor wrote: > > On Mon, Aug 15, 2022 at 12:39 PM Masahiko Sawada > wrote: > > > > On Fri, Jul 22, 2022 at 10:43 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Jul 19, 2022 at 1:30 PM John Naylor > &g

Re: Improve description of XLOG_RUNNING_XACTS

2022-09-15 Thread Masahiko Sawada
On Wed, Sep 14, 2022 at 6:33 PM Amit Kapila wrote: > > On Fri, Sep 9, 2022 at 6:18 AM Masahiko Sawada wrote: > > > > Updated the patch accordingly. > > > > I have created two xacts each with savepoints and after your patch, > the record will show xacts/subxac

Re: Introduce wait_for_subscription_sync for TAP tests

2022-09-09 Thread Masahiko Sawada
On Sat, Sep 10, 2022 at 6:45 AM Tom Lane wrote: > > Masahiko Sawada writes: > > On Fri, Sep 9, 2022 at 11:31 PM Tom Lane wrote: > >> Recently a number of buildfarm animals have failed at the same > >> place in src/test/subscription/t/100_bugs.pl [1][2][3][4]: >

Re: Introduce wait_for_subscription_sync for TAP tests

2022-09-09 Thread Masahiko Sawada
9 09:30:19.631 EDT [631b3feb.26e8:2] ERROR: error while shutting down streaming COPY: ERROR: could not find record while sending logically-decoded data: missing contrecord at 0/1D4FFF8 It's likely that the commit f6c5edb8abcac04eb3eac6da356e59d399b2bcef is relevant. Regards, -- Masahiko Sawada

Re: Improve description of XLOG_RUNNING_XACTS

2022-09-08 Thread Masahiko Sawada
On Tue, Aug 23, 2022 at 11:53 AM Kyotaro Horiguchi wrote: > > At Mon, 15 Aug 2022 11:16:56 +0900, Masahiko Sawada > wrote in > > Sorry for the late reply. > > No worries. Anyway I was in a long (as a Japanese:) vacation. > > > On Thu, Jul 28, 2022 at 4:29 P

Re: pgsql: Add ALTER SUBSCRIPTION ... SKIP.

2022-09-06 Thread Masahiko Sawada
then we can > think of replacing "with" in the above messages with "finished at". I > see your point related to using "finished at" for PREPARE may not be a > good idea but I don't have better ideas for the same. Given that the user normally doesn't need to be aware of the difference between start LSN and end LSN in the context of using this feature, I think we can use "with LSN %X/%X". Regards, -- Masahiko Sawada

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-08-27 Thread Masahiko Sawada
On Sat, Aug 27, 2022 at 7:24 PM Amit Kapila wrote: > > On Sat, Aug 27, 2022 at 1:06 PM Masahiko Sawada wrote: > > > > On Sat, Aug 27, 2022 at 3:56 PM Amit Kapila wrote: > > > > > > On Fri, Jul 29, 2022 at 12:15 PM Amit Kapila > > > wrote: >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-08-27 Thread Masahiko Sawada
clusion. Since we took this approach only on the master the back branches are not affected. The new test scenario makes sense to me and looks better than the one I have. Regarding the fix, I think we should use TransactionIdFollowsOrEquals() instead of NormalTransactionIdPrecedes(): + for (off = 0; off < b

Re: pg15b3: crash in paralell vacuum

2022-08-18 Thread Masahiko Sawada
m_pages = 0, estimated_count = false, num_index_tuples = 0, > tuples_removed = 0, pages_newly_deleted = 0, pages_deleted = 1, > pages_free = 0}} The status = 11 is invalid value. Probably because indstats was not initialized to 0 as I mentioned. Justin, if it's reproducible in your env

Re: pg15b3: crash in paralell vacuum

2022-08-18 Thread Masahiko Sawada
On Thu, Aug 18, 2022 at 11:06 PM Masahiko Sawada wrote: > > Hi, > > On Thu, Aug 18, 2022 at 10:34 PM Justin Pryzby wrote: > > > > Immediately after upgrading an internal instance, a loop around "vacuum" did > > this: > > Thank you for the report! >

Re: pg15b3: crash in paralell vacuum

2022-08-18 Thread Masahiko Sawada
vinced yet but it could be a culprit that we missed doing memset(0) for the shared array of PVIndStats in parallel_vacuum_init(). This shared array was introduced in PG15. [1] https://github.com/postgres/postgres/blob/REL_15_STABLE/src/backend/access/heap/vacuumlazy.c#L2679 Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Logical WAL sender unresponsive during decoding commit

2022-08-16 Thread Masahiko Sawada
On Tue, Aug 16, 2022 at 2:32 PM Masahiko Sawada wrote: > > On Tue, Aug 16, 2022 at 2:31 PM Amit Kapila wrote: > > > > On Tue, Aug 16, 2022 at 10:56 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Aug 16, 2022 at 2:08 PM Amit Kapila > > >

Re: Logical WAL sender unresponsive during decoding commit

2022-08-15 Thread Masahiko Sawada
On Tue, Aug 16, 2022 at 2:31 PM Amit Kapila wrote: > > On Tue, Aug 16, 2022 at 10:56 AM Masahiko Sawada > wrote: > > > > On Tue, Aug 16, 2022 at 2:08 PM Amit Kapila wrote: > > > > > > On Tue, Aug 16, 2022 at 9:28 AM Andrey Borodin > > > wrote:

Re: Logical WAL sender unresponsive during decoding commit

2022-08-15 Thread Masahiko Sawada
kpatched. Regards, [1] https://www.postgresql.org/message-id/CAD21AoD%2BaNfLje%2B9JOqWbTiq1GL4BOp9_f7FxLADm8rS8cDhCQ%40mail.gmail.com -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-08-14 Thread Masahiko Sawada
On Fri, Jul 22, 2022 at 10:43 AM Masahiko Sawada wrote: > > On Tue, Jul 19, 2022 at 1:30 PM John Naylor > wrote: > > > > > > > > On Tue, Jul 19, 2022 at 9:11 AM Masahiko Sawada > > wrote: > > > > > I’d like to keep the first version simp

Re: Improve description of XLOG_RUNNING_XACTS

2022-08-14 Thread Masahiko Sawada
Sorry for the late reply. On Thu, Jul 28, 2022 at 4:29 PM Kyotaro Horiguchi wrote: > > At Thu, 28 Jul 2022 15:53:33 +0900, Masahiko Sawada > wrote in > > > > > Do you mean that both could be true at the same time? If I read > > GetRunningTransactionData() correctl

Re: bogus assert in logicalmsg_desc

2022-08-14 Thread Masahiko Sawada
This was added in 14, so we should > backpatch to that version. +1 The patch looks good to me. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-08-11 Thread Masahiko Sawada
On Thu, Aug 11, 2022 at 3:10 PM Amit Kapila wrote: > > On Mon, Aug 8, 2022 at 9:34 AM Amit Kapila wrote: > > > > On Wed, Aug 3, 2022 at 1:20 PM Masahiko Sawada > > wrote: > > > > > > > > > Oops, thanks for pointing it out. I've fixed it and

Re: optimize lookups in snapshot [sub]xip arrays

2022-08-09 Thread Masahiko Sawada
on-USE_SSE2 builds, there is no additional overhead as all assertion-related code in pg_lfind32 depends on USE_SSE2. > I'll look around to see if there are any > other suitable candidates. As you proposed, having a test module for that seems to be a good idea. We can add test codes for future

Re: Fix unmatched file identifications

2022-08-08 Thread Masahiko Sawada
On Tue, Aug 9, 2022 at 11:24 AM John Naylor wrote: > > On Tue, Aug 9, 2022 at 8:58 AM Masahiko Sawada wrote: > > I found that there are two .c and .h files whose identification in the > > header comment doesn't match its actual path. > > > The attached small patch fix

Re: optimize lookups in snapshot [sub]xip arrays

2022-08-08 Thread Masahiko Sawada
that. This is done in v9. > > > Aside from that, I plan to commit this > > week unless there is further bikeshedding. > > Great, thanks. The patch looks good to me. One minor point is: + * IDENTIFICATION + * src/port/pg_lfind.h The path doesn't match to the actual file path, src/include/port/pg_lfind.h. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Fix unmatched file identifications

2022-08-08 Thread Masahiko Sawada
/fe_utils/cancel.c has: * src/fe-utils/cancel.c * * The attached small patch fixes them. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ v1-0001-Fix-unmatched-file-identifications.patch Description

Re: Handle infinite recursion in logical replication setup

2022-08-04 Thread Masahiko Sawada
i-directional logical replication with two nodes, they may use origin = 'none'. Therefore, it seems to me that this feature works only for a narrow situation and has false positives. Since it has been the user's responsibility not to try to make the data inconsistent by the initial table sync, I think that it might be sufficient if we note the risk in the documentation. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: support for SSE2 intrinsics

2022-08-03 Thread Masahiko Sawada
the commit message: s/hepler/helper/ The rest looks good to me. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Introduce wait_for_subscription_sync for TAP tests

2022-08-03 Thread Masahiko Sawada
On Wed, Aug 3, 2022 at 1:51 PM Amit Kapila wrote: > > On Sat, Jul 30, 2022 at 12:25 PM Amit Kapila wrote: > > > > On Thu, Jul 28, 2022 at 6:37 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Jul 27, 2022 at 7:08 PM shiy.f...@fujitsu.com > > >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-08-02 Thread Masahiko Sawada
On Tue, Aug 2, 2022 at 5:31 PM shiy.f...@fujitsu.com wrote: > > On Mon, Aug 1, 2022 10:31 PM Amit Kapila wrote: > > > > On Mon, Aug 1, 2022 at 7:46 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Jul 29, 2022 at 3:45 PM Amit Kapila > > wrote: &

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-08-02 Thread Masahiko Sawada
le() before erroring out in SnapBuildSerialize(). I think it's better to keep it consistent with nearby codes in this patch. I think if we prefer the style of closing the file by ereport(ERROR), it should be done for all of them in a separate patch. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-28 Thread Masahiko Sawada
On Thu, Jul 28, 2022 at 8:57 PM Amit Kapila wrote: > > On Thu, Jul 28, 2022 at 3:23 PM Amit Kapila wrote: > > > > On Tue, Jul 26, 2022 at 1:22 PM Masahiko Sawada > > wrote: > > > > > > Okay, I've attached an updated patch that does the above idea.

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-28 Thread Masahiko Sawada
On Thu, Jul 28, 2022 at 4:13 PM Amit Kapila wrote: > > On Thu, Jul 28, 2022 at 11:56 AM Masahiko Sawada > wrote: > > > > On Thu, Jul 28, 2022 at 12:21 PM Amit Kapila > > wrote: > > > > > > On Thu, Jul 28, 2022 at 7:18 AM Masahiko Sawada > >

Re: Improve description of XLOG_RUNNING_XACTS

2022-07-28 Thread Masahiko Sawada
String(buf, "; subxid overflowed"); > Do you mean that both could be true at the same time? If I read GetRunningTransactionData() correctly, that doesn't happen. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-28 Thread Masahiko Sawada
() an On Thu, Jul 28, 2022 at 12:21 PM Amit Kapila wrote: > > On Thu, Jul 28, 2022 at 7:18 AM Masahiko Sawada wrote: > > > > On Wed, Jul 27, 2022 at 8:33 PM Amit Kapila wrote: > > > > > > > > I have changed accordingly in the attached > > >

Re: Improve description of XLOG_RUNNING_XACTS

2022-07-27 Thread Masahiko Sawada
On Thu, Jul 21, 2022 at 10:13 PM Ashutosh Bapat wrote: > > Hi > > On Thu, Jul 21, 2022 at 6:44 AM Masahiko Sawada wrote: > > > > Hi, > > > > I realized that standby_desc_running_xacts() in standbydesc.c doesn't > > describe subtransactio

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-27 Thread Masahiko Sawada
On Wed, Jul 27, 2022 at 8:33 PM Amit Kapila wrote: > > On Mon, Jul 25, 2022 at 11:26 AM Masahiko Sawada > wrote: > > > > On Mon, Jul 25, 2022 at 10:45 AM Masahiko Sawada > > wrote: > > > > I've attached the patch for REl15 that I forgot. > > &

Re: Introduce wait_for_subscription_sync for TAP tests

2022-07-27 Thread Masahiko Sawada
On Wed, Jul 27, 2022 at 8:54 PM Amit Kapila wrote: > > On Tue, Jul 26, 2022 at 1:12 PM Masahiko Sawada wrote: > > > > On Tue, Jul 26, 2022 at 2:01 PM Amit Kapila wrote: > > > > > > 2. > > > +# wait for the replication to catchup i

Re: Introduce wait_for_subscription_sync for TAP tests

2022-07-27 Thread Masahiko Sawada
On Wed, Jul 27, 2022 at 7:08 PM shiy.f...@fujitsu.com wrote: > > On Tue, Jul 26, 2022 3:42 PM Masahiko Sawada wrote: > > > > I've attached an updated patch as well as a patch to remove duplicated > > waits in 007_ddl.pl. > > > > Thanks for your patch

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-26 Thread Masahiko Sawada
On Tue, Jul 26, 2022 at 2:18 PM Amit Kapila wrote: > > On Tue, Jul 26, 2022 at 7:00 AM Masahiko Sawada wrote: > > > > On Mon, Jul 25, 2022 at 7:57 PM shiy.f...@fujitsu.com > > wrote: > > > > > > Hi, > > > > > > I did some performanc

Re: Introduce wait_for_subscription_sync for TAP tests

2022-07-26 Thread Masahiko Sawada
On Tue, Jul 26, 2022 at 2:01 PM Amit Kapila wrote: > > On Tue, Jul 26, 2022 at 7:07 AM Masahiko Sawada wrote: > > > > Hi, > > > > In tap tests for logical replication, we have the following code in many > > places: > > > > $node_publisher-&g

Introduce wait_for_subscription_sync for TAP tests

2022-07-25 Thread Masahiko Sawada
e data. The attached patch introduce a new function wait_for_subscription_sync(). With this function, we can replace the above code with this one function as follows: $node_subscriber->wait_for_subscription_sync($node_publisher, 'tap_sub'); Regards, -- Masahiko Sawada EDB: https://www.enterpri

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-25 Thread Masahiko Sawada
n from that function before doing bearch() if the parsed->xinfo doesn't have XACT_XINFO_HAS_INVALS. That would save calling bsearch() for non-catalog-modifying transactions. Is it worth trying? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-24 Thread Masahiko Sawada
On Mon, Jul 25, 2022 at 10:45 AM Masahiko Sawada wrote: > > On Sat, Jul 23, 2022 at 8:32 PM Amit Kapila wrote: > > > > On Fri, Jul 22, 2022 at 11:48 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Jul 20, 2022 at 5:50 PM Amit Kapila > > >

Re: [BUG] Logical replication failure "ERROR: could not map filenode "base/13237/442428" to relation OID" with catalog modifying txns

2022-07-22 Thread Masahiko Sawada
On Wed, Jul 20, 2022 at 5:50 PM Amit Kapila wrote: > > On Wed, Jul 20, 2022 at 1:28 PM Masahiko Sawada wrote: > > > > On Wed, Jul 20, 2022 at 2:19 PM Amit Kapila wrote: > > > > > > On Wed, Jul 20, 2022 at 9:01 AM Masahiko Sawada > > > wrote

Re: slot_creation_error failures

2022-07-21 Thread Masahiko Sawada
ror' - -slot_name|slot_type|active --+-+-- -(0 rows) - Regards, [1] https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=fairywren=2022-04-11%2021%3A04%3A15=test-decoding-check -- Masahiko Sawada EDB: https://www.enterprisedb.com/

<    2   3   4   5   6   7   8   9   10   11   >