Re: Pgoutput not capturing the generated columns

2024-09-20 Thread Masahiko Sawada
On Thu, Sep 19, 2024 at 9:26 PM Amit Kapila wrote: > > On Fri, Sep 20, 2024 at 4:16 AM Peter Smith wrote: > > > > On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada > > wrote: > > > > > > On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila > >

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-20 Thread Masahiko Sawada
On Thu, Sep 19, 2024 at 10:46 PM Amit Kapila wrote: > > On Fri, Sep 20, 2024 at 5:13 AM David Rowley wrote: > > > > On Fri, 20 Sept 2024 at 05:03, Masahiko Sawada > > wrote: > > > I've done other benchmarking tests while changing the memory block >

Re: Conflict detection for update_deleted in logical replication

2024-09-19 Thread Masahiko Sawada
On Tue, Sep 17, 2024 at 9:29 PM Amit Kapila wrote: > > On Tue, Sep 17, 2024 at 11:24 PM Masahiko Sawada > wrote: > > > > On Mon, Sep 16, 2024 at 11:53 PM Amit Kapila > > wrote: > > > > > > On Tue, Sep 17, 2024 at 6:08 AM Masahiko Sawada > &g

Re: Pgoutput not capturing the generated columns

2024-09-19 Thread Masahiko Sawada
On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila wrote: > > On Tue, Sep 17, 2024 at 12:04 PM Peter Smith wrote: > > > > On Tue, Sep 17, 2024 at 4:15 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Sep 16, 2024 at 8:09 PM Peter Smith wrote:

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-19 Thread Masahiko Sawada
appreciated if you also do similar tests and share the results. Regards, [1] https://www.postgresql.org/message-id/CAD21AoAaN4jaJ%3DW%2BWprHvc0cGCf80SkiFQhRc6R%2BX3-05HAFqw%40mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-19 Thread Masahiko Sawada
On Wed, Sep 18, 2024 at 8:55 PM Amit Kapila wrote: > > On Thu, Sep 19, 2024 at 6:46 AM David Rowley wrote: > > > > On Thu, 19 Sept 2024 at 11:54, Masahiko Sawada > > wrote: > > > I've done some benchmark tests for three different code bases with > >

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-18 Thread Masahiko Sawada
On Tue, Sep 17, 2024 at 2:06 AM Amit Kapila wrote: > > On Mon, Sep 16, 2024 at 10:43 PM Masahiko Sawada > wrote: > > > > On Fri, Sep 13, 2024 at 3:58 AM Amit Kapila wrote: > > > > > > Can we try reducing the size of > > > 8MB memory bl

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-17 Thread Masahiko Sawada
On Mon, Sep 16, 2024 at 9:24 AM Noah Misch wrote: > > On Thu, Sep 12, 2024 at 03:42:48PM -0700, Masahiko Sawada wrote: > > On Tue, Sep 10, 2024 at 3:05 PM Noah Misch wrote: > > > On Tue, Sep 10, 2024 at 05:56:47PM -0400, Tom Lane wrote: > > > > Got it. So now

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-17 Thread Masahiko Sawada
On Tue, Sep 17, 2024 at 2:06 AM Amit Kapila wrote: > > On Mon, Sep 16, 2024 at 10:43 PM Masahiko Sawada > wrote: > > > > On Fri, Sep 13, 2024 at 3:58 AM Amit Kapila wrote: > > > > > > On Thu, Sep 12, 2024 at 4:03 AM Masahiko Sawada > > > wro

Re: Conflict detection for update_deleted in logical replication

2024-09-17 Thread Masahiko Sawada
On Mon, Sep 16, 2024 at 11:53 PM Amit Kapila wrote: > > On Tue, Sep 17, 2024 at 6:08 AM Masahiko Sawada wrote: > > > > On Fri, Sep 13, 2024 at 12:56 AM shveta malik > > wrote: > > > > > > On Fri, Sep 13, 2024 at 11:38 AM Amit Kapila > > >

Re: Pgoutput not capturing the generated columns

2024-09-16 Thread Masahiko Sawada
an error (as we do today)? or always > > send NULL? > > For this scenario, I suggested (see [1] #3) that the code could give a > WARNING. As I wrote up-thread: This combination doesn't seem > like something a user would do intentionally, so just silently > ignoring it (which the current patch does) is likely going to give > someone unexpected results/grief. It gives a WARNING, and then publishes the specified generated column data (even if publish_generated_column = false)? If so, it would mean that specifying the generated column to the column list means to publish its data regardless of the publish_generated_column parameter value. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Conflict detection for update_deleted in logical replication

2024-09-16 Thread Masahiko Sawada
nt to > handle replication delays caused by network lag or other factors, > assuming clock skew has already been addressed. I think that in a non-bidirectional case the value could need to be a large number. Is that right? Regards, [1] https://www.postgresql.org/message-id/20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-09-16 Thread Masahiko Sawada
users specify a generated column to the column list and set publish_generated_column = false, in the first place? raise an error (as we do today)? or always send NULL? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-16 Thread Masahiko Sawada
On Fri, Sep 13, 2024 at 3:58 AM Amit Kapila wrote: > > On Thu, Sep 12, 2024 at 4:03 AM Masahiko Sawada wrote: > > > > We have several reports that logical decoding uses memory much more > > than logical_decoding_work_mem[1][2][3]. For instance in one of the > > rep

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-12 Thread Masahiko Sawada
On Tue, Sep 10, 2024 at 3:05 PM Noah Misch wrote: > > On Tue, Sep 10, 2024 at 05:56:47PM -0400, Tom Lane wrote: > > Masahiko Sawada writes: > > > On Tue, Sep 10, 2024 at 11:57 AM Tom Lane wrote: > > >> Yeah, that seems like it could work. But are we sure that

Using per-transaction memory contexts for storing decoded tuples

2024-09-11 Thread Masahiko Sawada
ad. Regards, [1] https://www.postgresql.org/message-id/CAMnUB3oYugXCBLSkih%2BqNsWQPciEwos6g_AMbnz_peNoxfHwyw%40mail.gmail.com [2] https://www.postgresql.org/message-id/17974-f8c9d353a62f414d%40postgresql.org [3] https://www.postgresql.org/message-id/DB9PR07MB71808AC6C7770AF2FB36B95BCB252%40DB9

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-10 Thread Masahiko Sawada
On Tue, Sep 10, 2024 at 11:57 AM Tom Lane wrote: > > Masahiko Sawada writes: > > An alternative way would be that we store the char signedness in the > > control file, and gin_trgm_ops opclass reads it if the bytes in the > > meta page shows 'unset'. The c

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-10 Thread Masahiko Sawada
On Mon, Sep 9, 2024 at 11:25 PM Tom Lane wrote: > > Masahiko Sawada writes: > > On Mon, Sep 9, 2024 at 4:42 PM Tom Lane wrote: > >> Do you have an idea for how we'd get > >> this to happen during pg_upgrade, exactly? > > > What I was thinking is th

Re: First draft of PG 17 release notes

2024-09-10 Thread Masahiko Sawada
ser that is already mentioned in the release note would fall in this type too; it's not a feature aimed just for extension authors, but it's kind of source and internal changes IMO. Since the DSM registry feature is described in the doc, I think it would make sense to have it in the release no

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-09 Thread Masahiko Sawada
On Mon, Sep 9, 2024 at 4:42 PM Tom Lane wrote: > > Masahiko Sawada writes: > > When do we set the byte on the primary server? If it's the first time > > to use the GIN index, secondary servers would have to wait for the > > primary to use the GIN index, which coul

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-09 Thread Masahiko Sawada
would have to wait for the primary to use the GIN index, which could be an unpredictable time or it may never come depending on index usages. Probably we can make pg_upgrade set the byte in the meta page of GIN indexes that use the gin_trgm_ops. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-09-09 Thread Masahiko Sawada
On Mon, Sep 9, 2024 at 2:38 AM Shubham Khanna wrote: > > On Thu, Aug 29, 2024 at 11:46 AM Amit Kapila wrote: > > > > On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila > > > wro

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-06 Thread Masahiko Sawada
On Fri, Aug 30, 2024 at 8:10 PM Noah Misch wrote: > > On Thu, Aug 29, 2024 at 03:48:53PM -0500, Masahiko Sawada wrote: > > On Sun, May 19, 2024 at 6:46 AM Noah Misch wrote: > > > If I were standardizing pg_trgm on one or the other notion of "char", I > > &g

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-08-29 Thread Masahiko Sawada
On Sun, May 19, 2024 at 6:46 AM Noah Misch wrote: > > On Wed, May 15, 2024 at 02:56:54PM +0900, Masahiko Sawada wrote: > > > > How about extending amcheck to support GIN and GIst indexes so that it > > can detect potential data incompatibility due to changing 'ch

Re: Pgoutput not capturing the generated columns

2024-08-28 Thread Masahiko Sawada
On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila wrote: > > On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada wrote: > > > > On Wed, May 8, 2024 at 4:14 PM Shubham Khanna > > wrote: > > > > > > On Wed, May 8, 2024 at 11:39 AM Rajendra Kumar Dangwal &

Re: Fix memory counter update in reorderbuffer

2024-08-26 Thread Masahiko Sawada
On Mon, Aug 26, 2024 at 2:27 PM Nathan Bossart wrote: > > On Mon, Aug 26, 2024 at 11:25:53AM -0700, Masahiko Sawada wrote: > > Thank you for reviewing the patch. Pushed. > > nitpick: I think this one needs a pgindent run. Thank you for pointing it out. I forgot to check w

Re: Fix memory counter update in reorderbuffer

2024-08-26 Thread Masahiko Sawada
On Sun, Aug 25, 2024 at 9:29 PM Amit Kapila wrote: > > On Fri, Aug 23, 2024 at 3:44 PM Masahiko Sawada wrote: > > > > On Tue, Aug 20, 2024 at 9:29 PM Amit Kapila wrote: > > > > > > > I've updated the updated patch with regression tests. > > &g

Re: Fix memory counter update in reorderbuffer

2024-08-23 Thread Masahiko Sawada
On Tue, Aug 20, 2024 at 9:29 PM Amit Kapila wrote: > > On Tue, Aug 20, 2024 at 5:55 PM Masahiko Sawada wrote: > > > > On Fri, Aug 16, 2024 at 12:22 AM Shlok Kyal > > wrote: > > > > > > > Thank you for testing the patch! > > > > I'

Re: Fix memory counter update in reorderbuffer

2024-08-20 Thread Masahiko Sawada
Hi, On Fri, Aug 16, 2024 at 12:22 AM Shlok Kyal wrote: > > On Wed, 7 Aug 2024 at 11:48, Amit Kapila wrote: > > > > On Wed, Aug 7, 2024 at 7:42 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Aug 6, 2024 at 1:12 PM Amit Kapila > > > wr

Re: Use pgBufferUsage for block reporting in analyze

2024-08-13 Thread Masahiko Sawada
On Fri, Aug 2, 2024 at 8:11 AM Masahiko Sawada wrote: > > On Wed, Jul 31, 2024 at 11:27 PM Anthonin Bonnefoy > wrote: > > > > On Wed, Jul 31, 2024 at 9:36 PM Masahiko Sawada > > wrote: > > > Meanwhile, I think we can push 0001 and 0002 patches since they a

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-08-12 Thread Masahiko Sawada
validation when replaying a RUNNING_XACTS record. Since a RUNNING_XACTS record has the latest XID on the primary, I think the startup process can compare it to the slot-xmin, and invalidate slots which are older than the age limit. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Fix memory counter update in reorderbuffer

2024-08-11 Thread Masahiko Sawada
On Sat, Aug 10, 2024 at 5:48 AM Masahiko Sawada wrote: > > On Fri, Aug 9, 2024 at 3:30 PM Amit Kapila wrote: > > > > On Thu, Aug 8, 2024 at 9:43 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Aug 7, 2024 at 3:17 PM Amit Kapila > > > wr

Re: Fix memory counter update in reorderbuffer

2024-08-09 Thread Masahiko Sawada
On Fri, Aug 9, 2024 at 3:30 PM Amit Kapila wrote: > > On Thu, Aug 8, 2024 at 9:43 PM Masahiko Sawada wrote: > > > > On Wed, Aug 7, 2024 at 3:17 PM Amit Kapila wrote: > > > > > > On Wed, Aug 7, 2024 at 7:42 AM Masahiko Sawada > > > wrote: > &

Re: Fix memory counter update in reorderbuffer

2024-08-08 Thread Masahiko Sawada
On Wed, Aug 7, 2024 at 3:17 PM Amit Kapila wrote: > > On Wed, Aug 7, 2024 at 7:42 AM Masahiko Sawada wrote: > > > > On Tue, Aug 6, 2024 at 1:12 PM Amit Kapila wrote: > > > > > > On Sat, Aug 3, 2024 at 1:21 AM Masahiko Sawada > > > wrote: > &

Re: Fix memory counter update in reorderbuffer

2024-08-06 Thread Masahiko Sawada
On Tue, Aug 6, 2024 at 1:12 PM Amit Kapila wrote: > > On Sat, Aug 3, 2024 at 1:21 AM Masahiko Sawada wrote: > > > > I found a bug in the memory counter update in reorderbuffer. It was > > introduced by commit 5bec1d6bc5e, so pg17 and master are affected. > > >

Fix memory counter update in reorderbuffer

2024-08-02 Thread Masahiko Sawada
the patch that fixes the problem and includes a test case (the test part might not be committed as it slows the test case). Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com fix_memory_counter_update_in_reorderbuffer.patch Description: Binary data

Re: Use pgBufferUsage for block reporting in analyze

2024-08-01 Thread Masahiko Sawada
On Wed, Jul 31, 2024 at 11:27 PM Anthonin Bonnefoy wrote: > > On Wed, Jul 31, 2024 at 9:36 PM Masahiko Sawada wrote: > > Meanwhile, I think we can push 0001 and 0002 patches since they are in > > good shape. I've updated commit messages to them and slightly changed

Re: Use pgBufferUsage for block reporting in analyze

2024-07-31 Thread Masahiko Sawada
they are in good shape. I've updated commit messages to them and slightly changed 0002 patch to write "finished analyzing of table \"%s.%s.%s\" instead of "analyze of table \"%s.%s.%s\". Also, regarding 0003 patch, what is the main reason why we want to add WAL

Re: xid_wraparound tests intermittent failure.

2024-07-30 Thread Masahiko Sawada
On Tue, Jul 30, 2024 at 3:29 AM Andrew Dunstan wrote: > > > On 2024-07-29 Mo 5:25 PM, Masahiko Sawada wrote: > > I've attached the patch. Could you please test if the patch fixes the > instability you observed? > > Since we turn off autovacuum on all three tests an

Re: long-standing data loss bug in initial sync of logical replication

2024-07-30 Thread Masahiko Sawada
are not sure if that is good so the other alternative we > can pursue is to distribute invalidations in logical decoding > infrastructure [1] which has its downsides. > > Thoughts? Thank you for summarizing the problem and solutions! I think it's worth trying the idea of distributing invalidation messages, and we will see if there could be overheads or any further obstacles. IIUC this approach would resolve another issue we discussed before too[1]. Regards, [1] https://www.postgresql.org/message-id/CAD21AoAenVqiMjpN-PvGHL1N9DWnHSq673bfgr6phmBUzx=k...@mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Use pgBufferUsage for block reporting in analyze

2024-07-29 Thread Masahiko Sawada
Hi, On Mon, Jul 29, 2024 at 12:12 AM Anthonin Bonnefoy wrote: > > On Sat, Jul 27, 2024 at 12:35 AM Masahiko Sawada > wrote: > > An alternative idea would > > be to pass StringInfo to AcquireSampleRowsFunc() so that callback can > > write its messages there. This is

Re: xid_wraparound tests intermittent failure.

2024-07-29 Thread Masahiko Sawada
On Sat, Jul 27, 2024 at 1:06 PM Andrew Dunstan wrote: > > > On 2024-07-26 Fr 1:46 PM, Masahiko Sawada wrote: > > On Thu, Jul 25, 2024 at 6:52 PM Andrew Dunstan wrote: > >> > >> On 2024-07-25 Th 3:40 PM, Masahiko Sawada wrote: > >> > >&g

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-28 Thread Masahiko Sawada
On Fri, Jul 26, 2024 at 1:27 PM Melanie Plageman wrote: > > On Mon, Jul 22, 2024 at 9:26 PM Masahiko Sawada wrote: > > > > + CREATE TABLE ${table1}(col1 int) > > + WITH (autovacuum_enabled=false, fillfactor=10); > > + INSERT INTO $table1 VALUES(7); >

Re: Use pgBufferUsage for block reporting in analyze

2024-07-26 Thread Masahiko Sawada
On Wed, Jul 24, 2024 at 1:58 AM Anthonin Bonnefoy wrote: > > On Mon, Jul 22, 2024 at 10:59 PM Masahiko Sawada > wrote: > > The first line would vary depending on whether an autovacuum worker or > > not. And the above suggestion includes a change of term "row&

Re: Parallel heap vacuum

2024-07-26 Thread Masahiko Sawada
ber of parallel workers as we don't want to launch many workers on the table where most pages are all-visible. Which might not work for other table AMs. I'm updating the patch to implement parallel heap vacuum and will share the updated patch. It might take time as it requires to implement shared iteration support in radx tree. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-26 Thread Masahiko Sawada
On Thu, Jul 25, 2024 at 6:52 PM Andrew Dunstan wrote: > > > On 2024-07-25 Th 3:40 PM, Masahiko Sawada wrote: > > On Thu, Jul 25, 2024 at 11:06 AM Masahiko Sawada > wrote: > > On Thu, Jul 25, 2024 at 10:56 AM Andrew Dunstan wrote: > > On 2024-07-23 Tu 6:59 PM,

Re: Detailed release notes

2024-07-26 Thread Masahiko Sawada
changes, and I believe many users would like to do that. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-25 Thread Masahiko Sawada
On Thu, Jul 25, 2024 at 11:06 AM Masahiko Sawada wrote: > > On Thu, Jul 25, 2024 at 10:56 AM Andrew Dunstan wrote: > > > > > > On 2024-07-23 Tu 6:59 PM, Masahiko Sawada wrote: > > > > See > > <https://bitbucket.org/adunstan/rotfang-fdw/downloads/x

Re: xid_wraparound tests intermittent failure.

2024-07-25 Thread Masahiko Sawada
On Thu, Jul 25, 2024 at 10:56 AM Andrew Dunstan wrote: > > > On 2024-07-23 Tu 6:59 PM, Masahiko Sawada wrote: > > See > <https://bitbucket.org/adunstan/rotfang-fdw/downloads/xid-wraparound-result.tar.bz2> > > > The failure logs are from a run where both tests

Re: xid_wraparound tests intermittent failure.

2024-07-23 Thread Masahiko Sawada
On Tue, Jul 23, 2024 at 3:49 AM Andrew Dunstan wrote: > > > On 2024-07-22 Mo 9:29 PM, Masahiko Sawada wrote: > > On Mon, Jul 22, 2024 at 12:53 PM Andrew Dunstan wrote: > > On 2024-07-22 Mo 12:46 PM, Tom Lane wrote: > > Masahiko Sawada writes: > > Looking at dod

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-23 Thread Masahiko Sawada
On Tue, Jul 23, 2024 at 5:43 AM Melanie Plageman wrote: > > On Mon, Jul 22, 2024 at 10:54 PM Masahiko Sawada > wrote: > > > > On Mon, Jul 22, 2024 at 6:26 PM Melanie Plageman > > wrote: > > > > > > On Mon, Jul 22, 2024 at 6:36 PM Tom Lane wr

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-22 Thread Masahiko Sawada
-- regardless > of how slowly it runs vacuum. Looking at copperhead's failure logs, I could not find that "VACUUM (VERBOSE, FREEZE) vac_horizon_floor_table;" wrote the number of index scans in logs. Is there any clue that made you think the test failed to do multiple index vacuum passes? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-22 Thread Masahiko Sawada
On Mon, Jul 22, 2024 at 12:53 PM Andrew Dunstan wrote: > > > On 2024-07-22 Mo 12:46 PM, Tom Lane wrote: > > Masahiko Sawada writes: > > Looking at dodo's failures, it seems that while it passes > module-xid_wraparound-check, all failures happened only during > tes

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-22 Thread Masahiko Sawada
urrent TidStore, we will have to re-adjust them if the TidStore changes in the future. So I think it's better and reliable to allow maintenance_work_mem to be a lower value or use injection points somehow. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Add on_error and log_verbosity options to file_fdw

2024-07-22 Thread Masahiko Sawada
FOREIGN TABLE > > > > =# table f_fdw_test_3 ; > > NOTICE: skipping row due to data type incompatibility at line 3 for > > column i: "a" > > NOTICE: 1 row was skipped due to data type incompatibility > >i | t > > ---+--- > >1 | a > >2 | b > > (2 rows) IIUC we have to execute ALTER FOREIGN TABLE to change the log_verbosity value and which requires to be the owner. Which seems not to be user-friendly. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Use pgBufferUsage for block reporting in analyze

2024-07-22 Thread Masahiko Sawada
row" to "tuple" for better consistency with VACUUM VERBOSE outputs. I think it would be great if autoanalyze also writes logs in the same format. IIUC with the patch, autoanalyze logs don't include the page and tuple statistics. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-22 Thread Masahiko Sawada
On Mon, Jul 22, 2024 at 9:46 AM Tom Lane wrote: > > Masahiko Sawada writes: > > Looking at dodo's failures, it seems that while it passes > > module-xid_wraparound-check, all failures happened only during > > testmodules-install-check-C. Can we check the s

Re: xid_wraparound tests intermittent failure.

2024-07-22 Thread Masahiko Sawada
test, "slow disk" could also be a reason. Looking at dodo's failures, it seems that while it passes module-xid_wraparound-check, all failures happened only during testmodules-install-check-C. Can we check the server logs written during xid_wraparound test in testmodules-install-check-C? I t

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-12 Thread Masahiko Sawada
On Thu, Jul 11, 2024 at 8:14 PM Fujii Masao wrote: > > > > On 2024/07/10 22:35, Masahiko Sawada wrote: > > BTW the new regression tests don't check the table and index names. > > Isn't it better to show table and index names for better > > diagnosability?

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-10 Thread Masahiko Sawada
On Wed, Jul 10, 2024 at 5:14 PM Masahiko Sawada wrote: > > On Wed, Jul 10, 2024 at 4:14 PM Fujii Masao > wrote: > > > > > > > > On 2024/07/10 12:13, Masahiko Sawada wrote: > > > On Sat, Jul 6, 2024 at 4:06 PM Fujii Masao > > > wrote: > &g

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-10 Thread Masahiko Sawada
On Wed, Jul 10, 2024 at 4:14 PM Fujii Masao wrote: > > > > On 2024/07/10 12:13, Masahiko Sawada wrote: > > On Sat, Jul 6, 2024 at 4:06 PM Fujii Masao > > wrote: > >> > >> Hi, > >> > >> I noticed that ALTER TABLE MERGE PARTITIONS and S

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-09 Thread Masahiko Sawada
le's tablespace, it is logical to set the parent table's tablespace as the merged table's tablespace. While the patch does not include test cases for SPLIT PARTITIONS, which is understandable as these commands use the common function that we have fixed, I believe it would be prudent to test SPLIT PARTITIONS as well since we could change it in the future development. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Parallel heap vacuum

2024-07-07 Thread Masahiko Sawada
On Fri, Jun 28, 2024 at 9:06 PM Amit Kapila wrote: > > On Fri, Jun 28, 2024 at 9:44 AM Masahiko Sawada wrote: > > > > # Benchmark results > > > > * Test-1: parallel heap scan on the table without indexes > > > > I created 20GB table, made garbage on th

Re: Parallel heap vacuum

2024-07-07 Thread Masahiko Sawada
attached the new version patch. BTW since we compute the number of parallel workers for the heap scan based on the table size, it's possible that we launch multiple workers even if most blocks are all-visible. It seems to be better if we calculate it based on (relpages - relallvisible). Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com parallel_heap_vacuum_scan_v2.patch Description: Binary data

Re: Conflict Detection and Resolution

2024-07-03 Thread Masahiko Sawada
On Mon, Jul 1, 2024 at 6:54 PM Amit Kapila wrote: > > On Mon, Jul 1, 2024 at 1:35 PM Masahiko Sawada wrote: > > > > Setting resolvers at table-level and subscription-level sounds good to > > me. DDLs for setting resolvers at subscription-level would need the > > sub

Re: Conflict Detection and Resolution

2024-07-01 Thread Masahiko Sawada
eason we are planning to keep resolvers at the > > table level. Here, I am asking to set resolvers at the subscription > > level rather than at the global level. > > Okay, got it. I misunderstood earlier that we want to replace table > level resolvers with subscription ones. > Having global configuration has one benefit that if the user has no > requirement to set different resolvers for different subscriptions or > tables, he may always set one global configuration and be done with > it. OTOH, I also agree with benefits coming with subscription level > configuration. Setting resolvers at table-level and subscription-level sounds good to me. DDLs for setting resolvers at subscription-level would need the subscription name to be specified? And another question is: a table-level resolver setting is precedent over all subscriber-level resolver settings in the database? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Conflict Detection and Resolution

2024-06-30 Thread Masahiko Sawada
because the commit timestamp of UPDATE is newer than DELETE' one.ard DELETE, because the commit timestamp of UPDATE is newer than DELETE' one. As a result, both nodes have the new version row. Regards, [1] https://www.enterprisedb.com/docs/pgd/latest/consistency/conflicts/#updatedelete-conflicts [2] https://docs.oracle.com/goldengate/c1230/gg-winux/GWUAD/configuring-conflict-detection-and-resolution.htm (see DELETEROWEXISTS conflict type) -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Parallel heap vacuum

2024-06-27 Thread Masahiko Sawada
e ideas but it might be applicable to our ART implementation. But I prefer to start with (2) since it would be easier. Feedback is very welcome. Regards, [1] https://db.in.tum.de/~leis/papers/artsync.pdf -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com diff --git a/src/backend/access/he

Re: Track the amount of time waiting due to cost_delay

2024-06-26 Thread Masahiko Sawada
add any extra pg_clock_gettime_ns() calls (as compare to v2). > Sounds good to me. I think it's better to keep the logic for throttling the reporting the delay message simple. It's an important consideration but executing parallel vacuum with delays would be less likely to be used in practice. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: New standby_slot_names GUC in PG 17

2024-06-26 Thread Masahiko Sawada
On Wed, Jun 26, 2024 at 6:15 PM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, June 26, 2024 12:49 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Wed, Jun 26, 2024 at 04:17:45AM +, Zhijie Hou (Fujitsu) wrote: > > > On Wednesday, June 2

Re: Vacuum statistics

2024-06-26 Thread Masahiko Sawada
is hard to predict, I think showing the high-water mark would help users to predict how much memory they set to maintenance_work_mem. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: New standby_slot_names GUC in PG 17

2024-06-25 Thread Masahiko Sawada
On Tue, Jun 25, 2024 at 5:32 PM Amit Kapila wrote: > > On Tue, Jun 25, 2024 at 12:30 PM Masahiko Sawada > wrote: > > > > On Tue, Jun 25, 2024 at 1:54 PM Amit Kapila wrote: > > > > > > > > So, my > > > preference is in order as follows: sy

Re: New standby_slot_names GUC in PG 17

2024-06-25 Thread Masahiko Sawada
On Tue, Jun 25, 2024 at 1:54 PM Amit Kapila wrote: > > On Tue, Jun 25, 2024 at 8:20 AM Masahiko Sawada wrote: > > > > On Tue, Jun 25, 2024 at 11:21 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > I agree the current name seems too generic and the sug

Re: New standby_slot_names GUC in PG 17

2024-06-24 Thread Masahiko Sawada
lots_on_standbys: it indicates that the standbys that enabled > slot sync should be listed in this GUC. > > logical_replication_wait_slots: it means the logical replication(logical > Walsender process) will wait for these slots to advance the confirm flush > lsn before proceeding. I fee

Re: suspicious valgrind reports about radixtree/tidstore on arm64

2024-06-20 Thread Masahiko Sawada
On Thu, Jun 20, 2024 at 4:58 PM John Naylor wrote: > > On Thu, Jun 20, 2024 at 8:12 AM Masahiko Sawada wrote: > > > On Thu, Jun 20, 2024 at 7:54 AM Tom Lane wrote: > > > > > > I don't know if there's any reason why the current order > > >

Re: suspicious valgrind reports about radixtree/tidstore on arm64

2024-06-19 Thread Masahiko Sawada
fix-proposed.patch does. I agree with radixtree-fix-proposed.patch. Even if we zero more fields in the node it would not add noticeable overheads. > > (The RT_NODE_48 case could be optimized a bit if we cared > to swap the order of its slot_idxs[] and isset[] arrays; > then the initial

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-18 Thread Masahiko Sawada
On Sat, Jun 15, 2024 at 8:47 PM Robert Treat wrote: > > On Thu, Jun 13, 2024 at 9:22 PM Masahiko Sawada wrote: > > On Fri, Jun 14, 2024 at 9:57 AM Masahiko Sawada > > wrote: > > > On Fri, Jun 14, 2024 at 9:41 AM Michael Paquier > > > wrote: > > &g

Re: Logical Replication of sequences

2024-06-17 Thread Masahiko Sawada
On Fri, Jun 14, 2024 at 4:04 PM Amit Kapila wrote: > > On Thu, Jun 13, 2024 at 6:14 PM Masahiko Sawada wrote: > > > > On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila wrote: > > > > > > On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada > > > wrote: &g

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-13 Thread Masahiko Sawada
On Fri, Jun 14, 2024 at 9:57 AM Masahiko Sawada wrote: > > On Fri, Jun 14, 2024 at 9:41 AM Michael Paquier wrote: > > > > On Thu, Jun 13, 2024 at 08:38:05PM -0400, Tom Lane wrote: > > > Masahiko Sawada writes: > > >> I was about to push the patch

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-13 Thread Masahiko Sawada
On Fri, Jun 14, 2024 at 9:41 AM Michael Paquier wrote: > > On Thu, Jun 13, 2024 at 08:38:05PM -0400, Tom Lane wrote: > > Masahiko Sawada writes: > >> I was about to push the patch but let me confirm just in case: is it > >> okay to bump the catversion even after

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-13 Thread Masahiko Sawada
On Tue, Jun 11, 2024 at 10:38 PM Masahiko Sawada wrote: > > On Fri, Jun 7, 2024 at 10:22 AM Masahiko Sawada wrote: > > > > On Wed, Jun 5, 2024 at 7:19 PM Andrey M. Borodin > > wrote: > > > > > > > > > > > > > On 4 Jun 2024,

Re: Logical Replication of sequences

2024-06-13 Thread Masahiko Sawada
On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila wrote: > > On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada wrote: > > > > On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila wrote: > > > > > > > > > Yeah, starting with a single worker sounds good for now. Do you

Re: Logical Replication of sequences

2024-06-13 Thread Masahiko Sawada
On Wed, Jun 12, 2024 at 6:59 PM Amit Kapila wrote: > > On Wed, Jun 12, 2024 at 10:44 AM Masahiko Sawada > wrote: > > > > On Tue, Jun 11, 2024 at 7:36 PM vignesh C wrote: > > > > > > 1) CREATE PUBLICATION syntax enhancement: > > > CREATE PUBLICAT

Re: Conflict Detection and Resolution

2024-06-12 Thread Masahiko Sawada
on details (table name, column name, conflict type, etc) to somewhere (e.g. a table or server logs) so that the users can resolve them manually. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Logical Replication of sequences

2024-06-11 Thread Masahiko Sawada
On Tue, Jun 11, 2024 at 7:36 PM vignesh C wrote: > > On Tue, 11 Jun 2024 at 12:38, Masahiko Sawada wrote: > > > > On Tue, Jun 11, 2024 at 12:25 PM vignesh C wrote: > > > > > > On Mon, 10 Jun 2024 at 14:48, Amit Kapila wrote: > > > > > >

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-11 Thread Masahiko Sawada
On Fri, Jun 7, 2024 at 10:22 AM Masahiko Sawada wrote: > > On Wed, Jun 5, 2024 at 7:19 PM Andrey M. Borodin wrote: > > > > > > > > > On 4 Jun 2024, at 00:26, Masahiko Sawada wrote: > > > > Thank you! Vacuum enhancement is a really good step forward

Re: Logical Replication of sequences

2024-06-11 Thread Masahiko Sawada
On Tue, Jun 11, 2024 at 12:25 PM vignesh C wrote: > > On Mon, 10 Jun 2024 at 14:48, Amit Kapila wrote: > > > > On Mon, Jun 10, 2024 at 12:43 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada > > > wr

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Masahiko Sawada
hecking vacuum delay effects. It doesn't mean to measure the impact of the changes on the vacuum duration, though. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Logical Replication of sequences

2024-06-10 Thread Masahiko Sawada
On Mon, Jun 10, 2024 at 3:14 PM Masahiko Sawada wrote: > > On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila wrote: > > > > On Fri, Jun 7, 2024 at 7:55 AM Masahiko Sawada > > wrote: > > > > > > On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila > > > wro

Re: Logical Replication of sequences

2024-06-09 Thread Masahiko Sawada
On Fri, Jun 7, 2024 at 7:30 PM Amit Kapila wrote: > > On Fri, Jun 7, 2024 at 7:55 AM Masahiko Sawada wrote: > > > > On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila wrote: > > > > > > On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada > > > wrote: > &

Re: Logical Replication of sequences

2024-06-06 Thread Masahiko Sawada
On Thu, Jun 6, 2024 at 6:40 PM Amit Kapila wrote: > > On Thu, Jun 6, 2024 at 11:10 AM Masahiko Sawada wrote: > > > > On Wed, Jun 5, 2024 at 9:30 PM Amit Kapila wrote: > > > > > > > > To achieve this, we can allow sequences to be copied during >

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-06 Thread Masahiko Sawada
On Wed, Jun 5, 2024 at 7:19 PM Andrey M. Borodin wrote: > > > > > On 4 Jun 2024, at 00:26, Masahiko Sawada wrote: > > Thank you! Vacuum enhancement is a really good step forward, and this small > change would help a lot of observability tools. > > > > On 4

Re: Logical Replication of sequences

2024-06-05 Thread Masahiko Sawada
REFRESH PUBLICATION WITH (copy_sequence = true); > > In addition to the above, the command Alter Subscription .. Refresh > Publication will fetch any missing sequences similar to what it does > for tables. On the subscriber side, do we need to track which sequences are created via CREATE/ALTER SUBSCRIPTION? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Logical Replication of sequences

2024-06-05 Thread Masahiko Sawada
e to the publisher, do the last upgrade preparation work (e.g. copying the latest sequence values if requried), and then change the application so that new transactions come to the subscriber. I remember the blog post about Knock doing a similar process to upgrade the clusters with minimal downtime[1]. Regards, [1] https://knock.app/blog/zero-downtime-postgres-upgrades -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-03 Thread Masahiko Sawada
s only for PG17. Feedback is very welcome. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com 0001-Revive-num_dead_tuples-column-of-pg_stat_progress_va.patch Description: Binary data

Re: First draft of PG 17 release notes

2024-05-21 Thread Masahiko Sawada
nt is 188, > which is similar to recent releases: > I found a typo: s/pg_statstatement/pg_stat_statement/ I've attached a patch to fix it. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com fix_pg_stat_statements.patch Description: Binary data

Re: PostgreSQL 17 Beta 1 release announcement draft

2024-05-20 Thread Masahiko Sawada
0mail.gmail.com [2] https://www.postgresql.org/message-id/CANWCAZYqWibTRCWs5mV57mLj1A0nbKX-eV5G%2Bd-KmBOGHTVY-w%40mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-05-20 Thread Masahiko Sawada
move this limitation from the copy command and then > add it as a subscription option later. > Thoughts? I think that if we want to support an option to replicate generated columns, the initial tablesync should support it too. Otherwise, we end up filling the target columns data with NULL during the initial tablesync but with replicated data during the streaming changes. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Lowering the minimum value for maintenance_work_mem

2024-05-19 Thread Masahiko Sawada
the minimum segment size of DSA, i.e. DSA_MIN_SEGMENT_SIZE. So we can lower the minimum maintenance_work_mem down to 256kB, from a vacuum perspective. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-05-14 Thread Masahiko Sawada
to support GIN and GIst indexes so that it can detect potential data incompatibility due to changing 'char' to 'unsigned char'? I think these new tests would be useful also for users to check if they really need to reindex indexes due to such changes. Also we fix pg_trgm so that it uses 'unsigned char' in PG18. Users who upgraded to PG18 can run the new amcheck tests on the primary as well as the standby. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

  1   2   3   4   5   6   7   8   9   10   >