Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-25 Thread Drouvot, Bertrand
Hi, On 4/25/23 6:43 AM, Amit Kapila wrote: On Mon, Apr 24, 2023 at 5:38 PM Drouvot, Bertrand wrote: We are waiting here at a maximum for 10 * $default_timeout (means 3 minutes) before we time out. Would you prefer to wait more than 3 minutes at a maximum? No, because I don't know what

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-25 Thread Drouvot, Bertrand
Hi, On 4/25/23 6:23 AM, Amit Kapila wrote: On Mon, Apr 24, 2023 at 3:36 PM Drouvot, Bertrand wrote: Without the second "pg_log_standby_snapshot()" then wait_for_subscription_sync() would be waiting some time on the poll for "SELECT count(1) = 0 FROM pg_subscription_rel WH

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-24 Thread Drouvot, Bertrand
Hi, On 4/24/23 11:45 AM, Amit Kapila wrote: On Mon, Apr 24, 2023 at 11:54 AM Amit Kapila wrote: On Mon, Apr 24, 2023 at 11:24 AM Drouvot, Bertrand wrote: Few comments: +# We can not test if the WAL file still exists immediately. +# We need to let some time to the standby

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-24 Thread Drouvot, Bertrand
Hi, On 4/24/23 8:24 AM, Amit Kapila wrote: On Mon, Apr 24, 2023 at 11:24 AM Drouvot, Bertrand wrote: Few comments: Thanks for looking at it! 1. +$node_subscriber->init(allows_streaming => 'logical'); +$node_subscriber->append_conf('postgresql.conf', 'max_replicat

Re: Autogenerate some wait events code and documentation

2023-04-24 Thread Drouvot, Bertrand
Hi, On 4/24/23 5:15 AM, Michael Paquier wrote: On Sat, Apr 22, 2023 at 03:36:05PM +0200, Drouvot, Bertrand wrote: On 4/20/23 3:09 AM, Michael Paquier wrote: It is clear that the format of the file is: - category - C symbol in enums. - Format in the system views. - Description in the docs

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-23 Thread Drouvot, Bertrand
Hi, On 4/24/23 6:04 AM, vignesh C wrote: On Wed, 12 Apr 2023 at 21:45, Drouvot, Bertrand wrote: hi hackers, In the logical decoding on standby thread [1], Andres proposed 2 new tests (that I did not find the time to complete before the finish line): - Test that we can subscribe

Re: Autogenerate some wait events code and documentation

2023-04-22 Thread Drouvot, Bertrand
Hi, On 4/20/23 3:09 AM, Michael Paquier wrote: On Wed, Mar 29, 2023 at 02:51:27PM +0200, Drouvot, Bertrand wrote: Just realized that more polishing was needed. Done in V2 attached. That would be pretty cool to get that done in an automated way, I've wanted that for a few years now. And I

Re: Synchronizing slots from primary to standby

2023-04-17 Thread Drouvot, Bertrand
Hi, On 4/14/23 3:22 PM, Drouvot, Bertrand wrote: Now that the "Minimal logical decoding on standby" patch series (mentioned up-thread) has been committed, I think we can resume working on this one ("Synchronizing slots from primary to standby"). I'll work on a rebase an

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-17 Thread Drouvot, Bertrand
Hi, On 4/17/23 11:55 AM, Alvaro Herrera wrote: On 2023-Apr-12, Drouvot, Bertrand wrote: I'm not sure if adding those 2 tests should be considered as an open item. I can add this open item if we think that makes sense. I'd be happy to do so but it looks like I don't have the privileges to edit

Re: Synchronizing slots from primary to standby

2023-04-14 Thread Drouvot, Bertrand
Hi, On 11/15/22 10:02 AM, Drouvot, Bertrand wrote: Hi, On 2/11/22 3:26 PM, Peter Eisentraut wrote: On 10.02.22 22:47, Bruce Momjian wrote: On Tue, Feb  8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote: Which means that if e.g. the standby_slot_names GUC differs from synchronize_slot_names

Re: Clean up hba.c of code freeing regexps

2023-04-13 Thread Drouvot, Bertrand
Hi, On 4/13/23 5:48 AM, Michael Paquier wrote: On Wed, Apr 12, 2023 at 10:25:42PM -0400, Tom Lane wrote: +1 for cleanup, if this is new code. It does us no good in the long run for v16 to handle this differently from both earlier and later versions. Okidoki. Let me know if anybody has an

Re: Add ps display while waiting for wal in read_local_xlog_page_guts

2023-04-13 Thread Drouvot, Bertrand
Hi, On 4/13/23 4:29 AM, Tom Lane wrote: sirisha chamarthi writes: pg_create_logical_replication_slot can take longer than usual on a standby when there is no activity on the primary. We don't have enough information in the pg_stat_activity or process title to debug why this is taking so long.

Re: Add ps display while waiting for wal in read_local_xlog_page_guts

2023-04-13 Thread Drouvot, Bertrand
Hi, On 4/13/23 12:43 AM, sirisha chamarthi wrote: Hi, pg_create_logical_replication_slot can take longer than usual on a standby when there is no activity on the primary. We don't have enough information in the pg_stat_activity or process title to debug why this is taking so long. Attached

Add two missing tests in 035_standby_logical_decoding.pl

2023-04-12 Thread Drouvot, Bertrand
hi hackers, In the logical decoding on standby thread [1], Andres proposed 2 new tests (that I did not find the time to complete before the finish line): - Test that we can subscribe to the standby (with the publication created on the primary) - Verify that invalidated logical slots do not

Missing wait_for_replay_catchup in 035_standby_logical_decoding.pl

2023-04-11 Thread Drouvot, Bertrand
hi hackers, while working on the issue reported by Noah in [1], I realized that there is an issue in 035_standby_logical_decoding.pl. The issue is here: " $node_standby->reload; $node_primary->psql('postgres', q[CREATE DATABASE testdb]); $node_primary->safe_psql('testdb', qq[CREATE TABLE

Re: Minimal logical decoding on standbys

2023-04-11 Thread Drouvot, Bertrand
Hi, On 4/11/23 10:55 AM, Drouvot, Bertrand wrote: Hi, I think we might want to add: $node_primary->wait_for_replay_catchup($node_standby); before calling the slot creation. It's done in the attached, would it be possible to give it a try please? Actually, let's also wait for the cascad

Re: Minimal logical decoding on standbys

2023-04-11 Thread Drouvot, Bertrand
Hi, On 4/11/23 10:20 AM, Drouvot, Bertrand wrote: Hi, On 4/11/23 7:36 AM, Noah Misch wrote: On Fri, Apr 07, 2023 at 11:12:26AM -0700, Andres Freund wrote: --- /dev/null +++ b/src/test/recovery/t/035_standby_logical_decoding.pl @@ -0,0 +1,720 @@ +# logical decoding on standby : test logical

Re: Minimal logical decoding on standbys

2023-04-11 Thread Drouvot, Bertrand
Hi, On 4/11/23 7:36 AM, Noah Misch wrote: On Fri, Apr 07, 2023 at 11:12:26AM -0700, Andres Freund wrote: --- /dev/null +++ b/src/test/recovery/t/035_standby_logical_decoding.pl @@ -0,0 +1,720 @@ +# logical decoding on standby : test logical decoding, +# recovery conflict and standby promotion.

Re: Minimal logical decoding on standbys

2023-04-07 Thread Drouvot, Bertrand
Hi, On 4/7/23 8:24 PM, Drouvot, Bertrand wrote: Hi, On 4/7/23 5:47 PM, Andres Freund wrote: Hi, - write a test that invalidated logical slots do not lead to retaining WAL I'm not sure how to do that since pg_switch_wal() and friends can't be executed on a standby. You can do

Re: Minimal logical decoding on standbys

2023-04-07 Thread Drouvot, Bertrand
Hi, On 4/7/23 8:27 PM, Drouvot, Bertrand wrote: Hi, I think some of the patches might have more reviewers than really applicable, and might also miss some. I'd appreciate if you could go over that... Sure, will do in a couple of hours. That looks good to me, just few remarks: 0005

Re: Minimal logical decoding on standbys

2023-04-07 Thread Drouvot, Bertrand
Hi, On 4/7/23 8:12 PM, Andres Freund wrote: Hi, On 2023-04-07 08:47:57 -0700, Andres Freund wrote: Integrated all of these. Here's my current version. Changes: - Integrated Bertrand's changes - polished commit messages of 0001-0003 - edited code comments for 0003, including

Re: Minimal logical decoding on standbys

2023-04-07 Thread Drouvot, Bertrand
Hi, On 4/7/23 5:47 PM, Andres Freund wrote: Hi, - write a test that invalidated logical slots do not lead to retaining WAL I'm not sure how to do that since pg_switch_wal() and friends can't be executed on a standby. You can do it on the primary and wait for the records to have been

Re: Minimal logical decoding on standbys

2023-04-07 Thread Drouvot, Bertrand
Hi, On 4/7/23 9:50 AM, Andres Freund wrote: Hi, Here's my current working state - I'll go to bed soon. Thanks a lot for this Andres! Changes: - shared catalog relations weren't handled correctly, because the dboid is InvalidOid for them. I wrote a test for that as well. -

Re: Minimal logical decoding on standbys

2023-04-07 Thread Drouvot, Bertrand
Hi, On 4/7/23 7:56 AM, Andres Freund wrote: Hi, On 2023-04-07 07:02:04 +0200, Drouvot, Bertrand wrote: Done in V63 attached and did change the associated comment a bit. Can you send your changes incrementally, relative to V62? I'm polishing them right now, and that'd make it a lot easier

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/7/23 5:47 AM, Amit Kapila wrote: On Thu, Apr 6, 2023 at 7:50 PM Drouvot, Bertrand wrote: Thanks! Will update 0005. I noticed a few typos in the latest patches. 0004 1. + * Physical walsenders don't need to be wakon up during replay unless Typo. Thanks! Fixed in V63 just

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/6/23 4:20 PM, Drouvot, Bertrand wrote: Hi, On 4/6/23 3:39 PM, Amit Kapila wrote: On Thu, Apr 6, 2023 at 6:32 PM Drouvot, Bertrand wrote: I don't think it could be possible to create logical walsenders on a standby if AllowCascadeReplication() is not true, or am I missing

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/7/23 4:18 AM, Andres Freund wrote: Hi, TBH, I don't like the state of 0001 much. I'm working on polishing it now. Thanks Andres! A lot of the new functions in slot.h don't seem right to me: - ObsoleteSlotIsInvalid() - isn't an obsolete slot by definition invalid? bad naming,

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
On 4/7/23 3:59 AM, Amit Kapila wrote: On Fri, Apr 7, 2023 at 6:55 AM Andres Freund wrote: On 2023-04-06 12:10:57 +0530, Amit Kapila wrote: After this, I think for backends that have active slots, it would simply cancel the current query. Will that be sufficient? Because we want the backend

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/6/23 3:39 PM, Amit Kapila wrote: On Thu, Apr 6, 2023 at 6:32 PM Drouvot, Bertrand wrote: I don't think it could be possible to create logical walsenders on a standby if AllowCascadeReplication() is not true, or am I missing something? Right, so why to even traverse walsenders

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/6/23 2:23 PM, Amit Kapila wrote: On Thu, Apr 6, 2023 at 11:29 AM Amit Kapila wrote: Thinking some more on this, I think such a slot won't decode any other records. During CreateInitDecodingContext->ReplicationSlotReserveWal, for standby's, we use lastReplayedEndRecPtr as restart_lsn.

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/6/23 7:59 AM, Amit Kapila wrote: On Wed, Apr 5, 2023 at 6:14 PM Drouvot, Bertrand wrote: On 4/5/23 12:28 PM, Amit Kapila wrote: On Wed, Apr 5, 2023 at 2:41 PM Drouvot, Bertrand wrote: Maybe we could change the doc with something among those lines instead? " Existing lo

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/6/23 11:55 AM, Amit Kapila wrote: On Thu, Apr 6, 2023 at 12:10 PM Amit Kapila wrote: On Wed, Apr 5, 2023 at 9:27 PM Drouvot, Bertrand wrote: Another comment on 0001. extern void CheckSlotRequirements(void); extern void CheckSlotPermissions(void); +extern void

Re: Minimal logical decoding on standbys

2023-04-06 Thread Drouvot, Bertrand
Hi, On 4/6/23 8:40 AM, Amit Kapila wrote: On Wed, Apr 5, 2023 at 9:27 PM Drouvot, Bertrand After this, I think for backends that have active slots, it would simply cancel the current query. Will that be sufficient? Because we want the backend process should exit and release the slot so

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/5/23 4:24 PM, Robert Haas wrote: On Tue, Apr 4, 2023 at 8:33 PM Jeff Davis wrote: For comments, I agree that WalSndWakeup() clearly needs a comment update. The call site in ApplyWalRecord() could also use a comment. You could add a comment at every call site, but I don't think that's

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/5/23 3:15 PM, Amit Kapila wrote: On Wed, Apr 5, 2023 at 6:14 PM Drouvot, Bertrand wrote: On 4/5/23 12:28 PM, Amit Kapila wrote: On Wed, Apr 5, 2023 at 2:41 PM Drouvot, Bertrand wrote: minor nitpick: + + /* Intentional fall through to session cancel */ + /* FALLTHROUGH */ Do we

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/5/23 1:59 PM, Amit Kapila wrote: On Wed, Apr 5, 2023 at 3:58 PM Amit Kapila wrote: On Wed, Apr 5, 2023 at 2:41 PM Drouvot, Bertrand wrote: minor nitpick: + + /* Intentional fall through to session cancel */ + /* FALLTHROUGH */ Do we need to repeat fall through twice in different

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/5/23 12:28 PM, Amit Kapila wrote: On Wed, Apr 5, 2023 at 2:41 PM Drouvot, Bertrand wrote: Maybe we could change the doc with something among those lines instead? " Existing logical slots on standby also get invalidated if wal_level on primary is reduced to less than 'lo

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/5/23 8:59 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 12:05 PM Amit Kapila wrote: On further thinking, as such this shouldn't be a problem because all the WAL records before PARAMETER_CHANGE record will have sufficient information so that they can get decoded. However, with the

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/5/23 2:33 AM, Jeff Davis wrote: On Tue, 2023-04-04 at 14:55 -0400, Robert Haas wrote: Thanks for your continued work on $SUBJECT. I just took a look at 0004, Thanks Robert for the feedback! and I think that at the very least the commit message needs work. Agree. Perhaps a

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/4/23 8:13 PM, Jeff Davis wrote: On Tue, 2023-04-04 at 11:42 +0200, Drouvot, Bertrand wrote: Done in V58 and now this is as simple as: Minor comments on 0004 (address if you agree): Thanks for the review! * Consider static inline for WalSndWakeupProcessRequests()? Agree

Re: Minimal logical decoding on standbys

2023-04-05 Thread Drouvot, Bertrand
Hi, On 4/4/23 7:53 PM, Andres Freund wrote: Hi, On 2023-04-04 18:54:33 +0200, Drouvot, Bertrand wrote: if (check_on_xid) { if (terminating) appendStringInfo(_msg, _("terminating process %d to release replication slot \"%s\" because it conflict

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 1:21 PM, Alvaro Herrera wrote: Hi, On 2023-Apr-03, Andres Freund wrote: Hm? That's what the _'s do. We build strings in parts in other places too. No, what _() does is mark each piece for translation separately. But a translation cannot be done on string pieces, and later

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 3:43 PM, Amit Kapila wrote: On Tue, Apr 4, 2023 at 6:05 PM Drouvot, Bertrand I think we might want to consider slot's effective_xmin instead of data.xmin as we use that to store xmin_horizon when we build the full snapshot. Oh, I did not know about the 'effective_xmin

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 1:43 PM, Amit Kapila wrote: On Tue, Apr 4, 2023 at 3:14 PM Drouvot, Bertrand wrote: +static inline bool +LogicalReplicationSlotXidsConflict(ReplicationSlot *s, TransactionId xid) +{ + TransactionId slot_xmin; + TransactionId slot_catalog_xmin; + + slot_xmin = s->data.x

Re: Patch proposal: New hooks in the connection path

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 12:08 AM, Gregory Stark (as CFM) wrote: This looks like it was a good discussion -- last summer. But it doesn't seem to be a patch under active development now. It sounds like there were some design constraints that still need some new ideas to solve and a new patch will be

Re: Split index and table statistics into different types of stats

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/3/23 11:47 PM, Gregory Stark (as CFM) wrote: On Thu, 16 Mar 2023 at 05:25, Drouvot, Bertrand wrote: My plan was to get [1] done before resuming working on the "Split index and table statistics into different types of stats" one. [1]: https://commitfest.postgresql.o

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 9:48 AM, Masahiko Sawada wrote: On Tue, Apr 4, 2023 at 10:55 AM Masahiko Sawada wrote: Regarding 0004 patch: @@ -2626,6 +2626,12 @@ InitWalSenderSlot(void) walsnd->sync_standby_priority = 0; walsnd->latch = >procLatch;

Re: Minimal logical decoding on standbys

2023-04-04 Thread Drouvot, Bertrand
Hi, On 4/4/23 7:57 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 8:51 PM Drouvot, Bertrand wrote: I made it as simple as: /* * If the slot is already invalid or is a non conflicting slot, we don't * need to do anything. */ islogical

Re: Minimal logical decoding on standbys

2023-04-03 Thread Drouvot, Bertrand
Hi, On 4/3/23 8:10 AM, Drouvot, Bertrand wrote: Hi, On 4/3/23 7:35 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 4:26 AM Jeff Davis wrote: Agreed, even Bertrand and myself discussed the same approach few emails above. BTW, if we have this selective logic to wake physical/logical walsenders

Re: Minimal logical decoding on standbys

2023-04-03 Thread Drouvot, Bertrand
+0200, Drouvot, Bertrand wrote: 5.3% doc/src/sgml/ 6.2% src/backend/access/transam/ 4.6% src/backend/replication/logical/ 55.6% src/backend/replication/ 4.4% src/backend/storage/ipc/ 6.9% src/backend/tcop/ 5.3% src/backend/ 3.8% src/include/catalog/ 5.3% src

Re: Pass heaprel to GlobalVisTestFor() in vacuumRedirectAndPlaceholder()

2023-04-03 Thread Drouvot, Bertrand
Hi, On 4/3/23 12:30 AM, Andres Freund wrote: Hi, On 2023-04-02 10:22:18 -0700, Peter Geoghegan wrote: On Sun, Apr 2, 2023 at 10:18 AM Peter Geoghegan wrote: Making nbtree page deletion more efficient when logical decoding is in use seems well worthwhile. There is an "XXX" comment about this

Re: Minimal logical decoding on standbys

2023-04-03 Thread Drouvot, Bertrand
Hi, On 4/3/23 7:35 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 4:26 AM Jeff Davis wrote: On Fri, 2023-03-31 at 02:50 -0700, Jeff Davis wrote: But if the ConditionVariableEventSleep() API is added, then I think we should change the non-recovery case to use a CV as well for consistency, and

Re: Minimal logical decoding on standbys

2023-04-03 Thread Drouvot, Bertrand
Hi, On 4/3/23 7:20 AM, Amit Kapila wrote: On Mon, Apr 3, 2023 at 1:31 AM Jeff Davis wrote: On Sun, 2023-04-02 at 10:11 +0200, Drouvot, Bertrand wrote: I was thinking that, if a new LogicalWalSndWakeup() replaces "ConditionVariableBroadcast(>replayedCV);" in ApplyWalRecord() t

Pass heaprel to GlobalVisTestFor() in vacuumRedirectAndPlaceholder()

2023-04-02 Thread Drouvot, Bertrand
hi hackers, now that the heap relation is passed down to vacuumRedirectAndPlaceholder() thanks to 61b313e47e, we can also pass it down to GlobalVisTestFor() too (to allow more pruning). Please find attached a tiny patch to do so. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS

Re: Minimal logical decoding on standbys

2023-04-02 Thread Drouvot, Bertrand
Hi, On 4/1/23 6:50 AM, Amit Kapila wrote: On Fri, Mar 31, 2023 at 7:14 PM Drouvot, Bertrand wrote: That sounds like a good idea. We could imagine creating a LogicalWalSndWakeup() doing the Walsender(s) triage based on a new variable (as you suggest). But, it looks to me that we: - would

Re: regression coverage gaps for gist and hash indexes

2023-03-31 Thread Drouvot, Bertrand
Hi, On 4/1/23 1:13 AM, Andres Freund wrote: Hi, On 2023-03-31 17:00:00 +0300, Alexander Lakhin wrote: 31.03.2023 15:55, Tom Lane wrote: See also the thread about bug #16329 [1]. Alexander promised to look into improving the test coverage in this area, maybe he can keep an eye on the WAL

Re: Minimal logical decoding on standbys

2023-03-31 Thread Drouvot, Bertrand
Hi, On 3/31/23 1:58 PM, Amit Kapila wrote: On Fri, Mar 31, 2023 at 4:17 PM Drouvot, Bertrand wrote: + * This is needed for logical decoding on standby. Indeed the "problem" is that + * WalSndWaitForWal() waits for the *replay* LSN to increase, but gets woken up + * by walreceive

Re: Autogenerate some wait events code and documentation

2023-03-29 Thread Drouvot, Bertrand
Hi, On 3/29/23 11:44 AM, Drouvot, Bertrand wrote: Looking forward to your feedback, Just realized that more polishing was needed. Done in V2 attached. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.comFrom

Autogenerate some wait events code and documentation

2023-03-29 Thread Drouvot, Bertrand
Hi hackers, In another thread [1], Thomas had the idea to $SUBJECT in a similar way to what is currently done with src/backend/storage/lmgr/lwlocknames.txt. Doing so, like in the attached patch proposal, would help to avoid: - wait event without documentation like observed in [2] - orphaned

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-28 Thread Drouvot, Bertrand
Hi, On 3/29/23 2:09 AM, Michael Paquier wrote: On Tue, Mar 28, 2023 at 05:43:26PM +0900, Kyotaro Horiguchi wrote: No. Fine by me, except that "block read requests" seems to suggest kernel read() calls, maybe because it's not clear whether "block" refers to our buffer blocks or file blocks to

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-27 Thread Drouvot, Bertrand
Hi, On 3/28/23 7:23 AM, Michael Paquier wrote: On Tue, Mar 28, 2023 at 12:36:15PM +0900, Kyotaro Horiguchi wrote: I found that commit ddfc2d9a37 removed the descriptions for pg_stat_get_blocks_fetched and pg_stat_get_blocks_hit. Right before that commit, monitoring.sgml had these lines: -

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-27 Thread Drouvot, Bertrand
On 3/28/23 12:41 AM, Michael Paquier wrote: On Mon, Mar 27, 2023 at 07:08:51PM +0900, Michael Paquier wrote: The patch has one mistake: PG_STAT_GET_XACT_FUNCENTRY_FLOAT8_MS does not need a slash on its last line or it would include the next, empty line. This could lead to mistakes (no need

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-27 Thread Drouvot, Bertrand
On 3/27/23 9:23 AM, Michael Paquier wrote: On Mon, Mar 27, 2023 at 08:54:13AM +0200, Drouvot, Bertrand wrote: Yes, something like V1 up-thread was doing. I think it can be added with your current proposal. Sure, I can write that. Or perhaps you'd prefer write something yourself? Please

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-27 Thread Drouvot, Bertrand
On 3/27/23 8:40 AM, Michael Paquier wrote: On Mon, Mar 27, 2023 at 07:45:26AM +0200, Drouvot, Bertrand wrote: Thanks! LGTM, but what about also taking care of pg_stat_get_xact_function_total_time() and pg_stat_get_xact_function_self_time() while at it? With a macro that uses

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-26 Thread Drouvot, Bertrand
Hi, On 3/27/23 3:20 AM, Michael Paquier wrote: On Sat, Mar 25, 2023 at 11:50:50AM +0900, Michael Paquier wrote: On Fri, Mar 24, 2023 at 06:58:30AM +0100, Drouvot, Bertrand wrote: - Does not include the refactoring for pg_stat_get_xact_function_total_time(), pg_stat_get_xact_function_self_time

Re: Orphaned wait event

2023-03-24 Thread Drouvot, Bertrand
Hi, On 3/23/23 11:00 PM, Thomas Munro wrote: I think if we want proper automation here we should look into a way to define wait events in a central file similar to what we do for src/backend/storage/lmgr/lwlocknames.txt. It could give the enum name, the display name, and the documentation

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-24 Thread Drouvot, Bertrand
Hi, On 3/24/23 1:04 AM, Michael Paquier wrote: On Thu, Mar 02, 2023 at 08:39:14AM +0100, Drouvot, Bertrand wrote: Yeah, there is some dependencies around this one. [1]: depends on it Current one depends of [2], [3] and [4] Waiting on Author is then the right state, thanks for having moved

Re: [BUG] pg_stat_statements and extended query protocol

2023-03-23 Thread Drouvot, Bertrand
Hi, On 3/22/23 10:35 PM, Imseih (AWS), Sami wrote: What about using an uint64 for calls? That seems more appropriate to me (even if queryDesc->totaltime->calls will be passed (which is int64), but that's already also the case for the "rows" argument and queryDesc->totaltime->rows_processed)

Re: Remove nonmeaningful prefixes in PgStat_* fields

2023-03-23 Thread Drouvot, Bertrand
Hi, On 3/23/23 1:09 AM, Michael Paquier wrote: On Wed, Mar 22, 2023 at 02:52:23PM -0400, Melanie Plageman wrote: This comment still has the t_ prefix as does the one for tuples_updated and deleted. otherwise, LGTM. Good catch. pgstat_count_heap_update() has a t_tuples_hot_updated, and

Re: [BUG] pg_stat_statements and extended query protocol

2023-03-22 Thread Drouvot, Bertrand
Hi, On 3/21/23 2:16 PM, Imseih (AWS), Sami wrote: This indeed feels a bit more natural seen from here, after looking at the code paths using an Instrumentation in the executor and explain, for example. At least, this stresses me much less than adding 16 bytes to EState for something restricted

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-22 Thread Drouvot, Bertrand
Hi, On 3/22/23 7:44 AM, Drouvot, Bertrand wrote: Hi, On 3/22/23 5:45 AM, Michael Paquier wrote: On Wed, Mar 22, 2023 at 11:37:03AM +0900, Kyotaro Horiguchi wrote: In the original description, "buffer fetches" appears to be a plural form of a compound noun and correct, similar to &q

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-22 Thread Drouvot, Bertrand
Hi, On 3/22/23 5:45 AM, Michael Paquier wrote: On Wed, Mar 22, 2023 at 11:37:03AM +0900, Kyotaro Horiguchi wrote: In the original description, "buffer fetches" appears to be a plural form of a compound noun and correct, similar to "buffer hits" mentioned later. If we reword it, I think it

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-20 Thread Drouvot, Bertrand
Hi, On 3/20/23 12:43 AM, Michael Paquier wrote: At the end, documenting both still sounds like the best move to me. Agree. Please find attached v1-0001-pg_stat_get_xact_blocks_fetched-and_hit-doc.patch doing so. I did not put the exact same wording as the one being removed in ddfc2d9, as:

Re: Remove nonmeaningful prefixes in PgStat_* fields

2023-03-20 Thread Drouvot, Bertrand
Hi, On 3/20/23 8:32 AM, Michael Paquier wrote: /* Total time previously charged to function, as of function start */ - instr_time save_f_total_time; + instr_time save_total_time; I have something to say about this one, though.. I find this change a bit

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-16 Thread Drouvot, Bertrand
On 3/16/23 12:46 PM, Michael Paquier wrote: On Thu, Mar 16, 2023 at 11:32:56AM +0100, Drouvot, Bertrand wrote: On 3/16/23 7:29 AM, Michael Paquier wrote: From what I get with this change, the number of tuples changed by DMLs have their computations done a bit earlier, Thanks for looking

Re: Speed-up shared buffers prewarming

2023-03-16 Thread Drouvot, Bertrand
Hi, On 3/15/23 10:40 PM, Matthias van de Meent wrote: On Wed, 15 Mar 2023 at 21:38, Konstantin Knizhnik wrote: Hi hackers, It is well known fact that queries using sequential scan can not be used to prewarm cache, because them are using ring buffer even if shared buffers are almost empty.

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-16 Thread Drouvot, Bertrand
Hi, On 3/16/23 7:29 AM, Michael Paquier wrote: On Mon, Mar 06, 2023 at 08:33:15AM +0100, Drouvot, Bertrand wrote: Thanks for having looked at it! Looking at that, I have a few comments. +tabentry = (PgStat_TableStatus *) entry_ref->pending; +tablestatus = palloc(siz

Re: Get rid of PgStat_BackendFunctionEntry

2023-03-16 Thread Drouvot, Bertrand
Hi, On 3/16/23 6:25 AM, Michael Paquier wrote: On Wed, Mar 08, 2023 at 01:38:38PM -0800, Nathan Bossart wrote: Looks reasonable to me. I have been catching up with this thread and the other thread, and indeed it looks like this is going to help in refactoring pgstatfuncs.c to have more

Re: [BUG] pg_stat_statements and extended query protocol

2023-03-13 Thread Drouvot, Bertrand
Hi, On 3/2/23 8:27 AM, Michael Paquier wrote: On Wed, Jan 25, 2023 at 11:22:04PM +, Imseih (AWS), Sami wrote: Doing some work with extended query protocol, I encountered the same issue that was discussed in [1]. It appears when a client is using extended query protocol and sends an Execute

Re: Minimal logical decoding on standbys

2023-03-10 Thread Drouvot, Bertrand
Hi, On 3/8/23 11:25 AM, Drouvot, Bertrand wrote: Hi, On 3/3/23 5:26 PM, Drouvot, Bertrand wrote: Hi, On 3/3/23 8:58 AM, Jeff Davis wrote: On Thu, 2023-03-02 at 11:45 -0800, Jeff Davis wrote: In this case it looks easier to add the right API than to be sure about whether it's needed

Re: Add shared buffer hits to pg_stat_io

2023-03-09 Thread Drouvot, Bertrand
Hi, On 3/9/23 2:23 PM, Melanie Plageman wrote: On Wed, Mar 8, 2023 at 2:23 PM Andres Freund wrote: On 2023-03-08 13:44:32 -0500, Melanie Plageman wrote: However, I am concerned that, while unlikely, this could be flakey. Something could happen to force all of those blocks out of shared

Re: Avoid multiple SetLatch() calls in procsignal_sigusr1_handler()

2023-03-09 Thread Drouvot, Bertrand
Hi, On 2/28/23 4:30 PM, Bharath Rupireddy wrote: Hi, Most of the multiplexed SIGUSR1 handlers are setting latch explicitly when the procsignal_sigusr1_handler() can do that for them at the end. Right. These multiplexed handlers are currently being used as SIGUSR1 handlers, not as

Re: Track IO times in pg_stat_io

2023-03-09 Thread Drouvot, Bertrand
Hi, On 3/9/23 1:34 AM, Andres Freund wrote: Hi, On 2023-03-08 12:55:34 +0100, Drouvot, Bertrand wrote: On 3/7/23 7:47 PM, Andres Freund wrote: On 2023-03-07 13:43:28 -0500, Melanie Plageman wrote: No, I don't think we can do that. It can be enabled on a per-session basis. Oh right. So it's

Re: Track IO times in pg_stat_io

2023-03-08 Thread Drouvot, Bertrand
Hi, On 3/7/23 7:47 PM, Andres Freund wrote: On 2023-03-07 13:43:28 -0500, Melanie Plageman wrote: Now I've a second thought: what do you think about resetting the related number of operations and *_time fields when enabling/disabling track_io_timing? (And mention it in the doc). That way

Re: Minimal logical decoding on standbys

2023-03-08 Thread Drouvot, Bertrand
Hi, On 3/3/23 5:26 PM, Drouvot, Bertrand wrote: Hi, On 3/3/23 8:58 AM, Jeff Davis wrote: On Thu, 2023-03-02 at 11:45 -0800, Jeff Davis wrote: In this case it looks easier to add the right API than to be sure about whether it's needed or not. I attached a sketch of one approach. Oh

Re: Track IO times in pg_stat_io

2023-03-07 Thread Drouvot, Bertrand
Hi, On 3/6/23 5:30 PM, Melanie Plageman wrote: Thanks for the review! On Tue, Feb 28, 2023 at 4:49 AM Drouvot, Bertrand wrote: On 2/26/23 5:03 PM, Melanie Plageman wrote: The timings will only be non-zero when track_io_timing is on That could lead to incorrect interpretation if one wants

Re: Add shared buffer hits to pg_stat_io

2023-03-07 Thread Drouvot, Bertrand
Hi, On 3/6/23 4:38 PM, Melanie Plageman wrote: Thanks for the review! On Tue, Feb 28, 2023 at 7:36 AM Drouvot, Bertrand wrote: BufferDesc * LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, -bool *foundPtr, IOContext

Re: Reconcile stats in find_tabstat_entry() and get rid of PgStat_BackendFunctionEntry

2023-03-05 Thread Drouvot, Bertrand
Hi, On 2/16/23 10:21 PM, Andres Freund wrote: Hi, On 2023-02-15 09:21:48 +0100, Drouvot, Bertrand wrote: diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index f793ac1516..b26e2a5a7a 100644 --- a/src/backend/utils/activity

Re: Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-03-04 Thread Drouvot, Bertrand
Hi, On 3/3/23 6:53 PM, Robert Haas wrote: On Fri, Mar 3, 2023 at 11:28 AM Drouvot, Bertrand wrote: Thanks for having looked at it! +1. Committed. Thanks! Not a big deal, but the commit message that has been used is not 100% accurate. Indeed, for gistxlogDelete, that's the other way

Re: Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-03-03 Thread Drouvot, Bertrand
Hi, On 3/3/23 12:30 PM, Amit Kapila wrote: On Thu, Mar 2, 2023 at 6:35 PM Drouvot, Bertrand wrote: On 1/6/23 11:05 AM, Drouvot, Bertrand wrote: Hi hackers, Please find attached a patch to $SUBJECT. The wrong comments have been discovered by Robert in [1]. Submitting this here

Re: Minimal logical decoding on standbys

2023-03-03 Thread Drouvot, Bertrand
Hi, On 3/3/23 8:58 AM, Jeff Davis wrote: On Thu, 2023-03-02 at 11:45 -0800, Jeff Davis wrote: In this case it looks easier to add the right API than to be sure about whether it's needed or not. I attached a sketch of one approach. Oh, that's very cool, thanks a lot! I'm not very

Re: Minimal logical decoding on standbys

2023-03-03 Thread Drouvot, Bertrand
Hi, On 3/2/23 8:45 PM, Jeff Davis wrote: On Thu, 2023-03-02 at 10:20 +0100, Drouvot, Bertrand wrote: Right, but in our case, right after the wakeup (the one due to the CV broadcast, aka the one that will remove it from the wait queue) we'll exit the loop due to: " /* check wh

Re: Fix comments in gistxlogDelete, xl_heap_freeze_page and xl_btree_delete

2023-03-02 Thread Drouvot, Bertrand
Hi, On 1/6/23 11:05 AM, Drouvot, Bertrand wrote: Hi hackers, Please find attached a patch to $SUBJECT. The wrong comments have been discovered by Robert in [1]. Submitting this here as a separate thread so it does not get lost in the logical decoding on standby thread. [1]: https

Re: Minimal logical decoding on standbys

2023-03-02 Thread Drouvot, Bertrand
Hi, On 3/2/23 1:40 AM, Jeff Davis wrote: On Wed, 2023-03-01 at 11:51 +0100, Drouvot, Bertrand wrote: Why not "simply" call ConditionVariablePrepareToSleep() without any call to ConditionVariableTimedSleep() later? ConditionVariableSleep() re-inserts itself into

Re: Generate pg_stat_get_xact*() functions with Macros

2023-03-01 Thread Drouvot, Bertrand
Hi, On 3/1/23 8:54 PM, Gregory Stark (as CFM) wrote: Looks like you have a path forward on this and it's not ready to commit yet? In which case I'll mark it Waiting on Author? Yeah, there is some dependencies around this one. [1]: depends on it Current one depends of [2], [3] and [4]

Re: Normalization of utility queries in pg_stat_statements

2023-03-01 Thread Drouvot, Bertrand
Hi, On 3/1/23 5:47 AM, Michael Paquier wrote: On Mon, Feb 20, 2023 at 11:34:59AM +0900, Michael Paquier wrote: With the patches.. Attached is an updated patch set, where I have done more refactoring work for the regression tests of pg_stat_statements, splitting pg_stat_statments.sql into the

Re: Minimal logical decoding on standbys

2023-03-01 Thread Drouvot, Bertrand
Hi, On 3/1/23 1:48 AM, Jeff Davis wrote: On Mon, 2023-02-27 at 09:40 +0100, Drouvot, Bertrand wrote: Please find attached V51 tiny rebase due to a6cd1fc692 (for 0001) and 8a8661828a (for 0005). [ Jumping into this thread late, so I apologize if these comments have already been covered

Re: Add shared buffer hits to pg_stat_io

2023-02-28 Thread Drouvot, Bertrand
Hi, On 2/25/23 9:16 PM, Melanie Plageman wrote: Hi, As suggested in [1], the attached patch adds shared buffer hits to pg_stat_io. Thanks for the patch! BufferDesc * LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, -bool

Re: Track IO times in pg_stat_io

2023-02-28 Thread Drouvot, Bertrand
Hi, On 2/26/23 5:03 PM, Melanie Plageman wrote: Hi, As suggested in [1], the attached patch adds IO times to pg_stat_io; Thanks for the patch! I started to have a look at it and figured out that a tiny rebase was needed (due to 728560db7d and b9f0e54bc9), so please find the rebase (aka V2)

Re: Change xl_hash_vacuum_one_page.ntuples from int to uint16

2023-02-27 Thread Drouvot, Bertrand
Hi, On 2/27/23 6:27 AM, Amit Kapila wrote: On Fri, Feb 17, 2023 at 7:44 PM Drouvot, Bertrand wrote: On 2/16/23 1:26 PM, Drouvot, Bertrand wrote: Hi, On 2/16/23 12:00 PM, Amit Kapila wrote: BTW, feel free to create the second patch (to align the types for variables/arguments

<    1   2   3   4   5   >