Re: Autogenerate some wait events code and documentation

2023-09-05 Thread Drouvot, Bertrand
Hi, On 9/6/23 5:44 AM, Michael Paquier wrote: On Wed, Sep 06, 2023 at 04:20:23AM +0200, Erik Wienhold wrote: FYI: https://en.wikipedia.org/wiki/Sentence_spacing That was an interesting read. Thanks. +1, thanks! Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source

Re: Autogenerate some wait events code and documentation

2023-09-05 Thread Drouvot, Bertrand
Hi, On 9/5/23 7:44 AM, Michael Paquier wrote: On Mon, Sep 04, 2023 at 02:14:58PM +0200, Drouvot, Bertrand wrote: # Build the descriptions. There are in camel-case. # LWLock and Lock classes do not need any modifications. Nit: 2 whitespaces before "There are in

Re: Autogenerate some wait events code and documentation

2023-09-04 Thread Drouvot, Bertrand
Hi, On 8/29/23 8:41 AM, Michael Paquier wrote: On Tue, Aug 29, 2023 at 08:17:10AM +0200, Drouvot, Bertrand wrote: That may be a bug in the matrix because of bb90022, as git am can be easily pissed. git am does not complain anymore. + # Generate the element name for the enums based

Re: Autogenerate some wait events code and documentation

2023-08-29 Thread Drouvot, Bertrand
Hi, On 8/28/23 10:04 AM, Michael Paquier wrote: On Mon, Jul 17, 2023 at 10:16:02AM +0900, Michael Paquier wrote: So you mean to switch a line that now looks like that: WAIT_EVENT_FOO_BAR FooBar"Waiting on Foo Bar." To that: FOO_BAR "Waiting on Foo Bar." Or even that:

Re: WIP: new system catalog pg_wait_event

2023-08-21 Thread Drouvot, Bertrand
Hi, On 8/20/23 10:07 AM, Michael Paquier wrote: On Sat, Aug 19, 2023 at 06:30:12PM +0200, Drouvot, Bertrand wrote: Thanks, fixed in v10. Okay. I have done an extra review of it, simplifying a few things in the function, the comments and the formatting, and applied the patch. Thanks

Re: WIP: new system catalog pg_wait_event

2023-08-19 Thread Drouvot, Bertrand
Hi, On 8/19/23 12:00 PM, Michael Paquier wrote: On Sat, Aug 19, 2023 at 11:35:01AM +0200, Drouvot, Bertrand wrote: Hi, On 8/18/23 12:31 PM, Michael Paquier wrote: Thanks! Please find attached v9 fixing this typo. I was looking at v8, and this looks pretty good to me. Great! I have

Re: WIP: new system catalog pg_wait_event

2023-08-19 Thread Drouvot, Bertrand
Hi, On 8/18/23 12:31 PM, Michael Paquier wrote: On Fri, Aug 18, 2023 at 10:56:55AM +0200, Drouvot, Bertrand wrote: Okay, using the plural form in v8 attached. Noting in passing: - Here is an example of how wait events can be viewed: + Here is are examples of how wait events can

Re: WIP: new system catalog pg_wait_event

2023-08-18 Thread Drouvot, Bertrand
Hi, On 8/18/23 12:37 AM, Michael Paquier wrote: On Thu, Aug 17, 2023 at 04:37:22PM +0900, Masahiro Ikeda wrote: On 2023-08-17 14:53, Drouvot, Bertrand wrote: BTW, is it better to start with "Waiting" like any other lines? For example, "Waiting for custom event \"worke

Re: WIP: new system catalog pg_wait_event

2023-08-18 Thread Drouvot, Bertrand
Hi, On 8/17/23 9:37 AM, Masahiro Ikeda wrote: Hi, On 2023-08-17 14:53, Drouvot, Bertrand wrote: The followings are additional comments for v7. 1) I am not sure that "pg_wait_event" is a good idea for the name if the new view.  How about "pg_wait_events" ins

Re: Synchronizing slots from primary to standby

2023-08-17 Thread Drouvot, Bertrand
Hi, On 8/14/23 11:52 AM, shveta malik wrote: We (myself and Ajin) performed the tests to compute the lag in standby slots as compared to primary slots with different number of slot-sync workers configured. Thanks! 3 DBs were created, each with 30 tables and each table having one

Re: WIP: new system catalog pg_wait_event

2023-08-16 Thread Drouvot, Bertrand
Hi, On 8/17/23 3:57 AM, Michael Paquier wrote: On Thu, Aug 17, 2023 at 10:53:02AM +0900, Masahiro Ikeda wrote: BTW, although I think this is outside the scope of this patch, it might be a good idea to be able to add a description to the API for custom wait events. Somebody on twitter has

Re: WIP: new system catalog pg_wait_event

2023-08-16 Thread Drouvot, Bertrand
Hi, On 8/17/23 3:53 AM, Masahiro Ikeda wrote: Hi, Thank you for creating the patch! I think it is a very useful view as a user. I will share some thoughts about the v6 patch. Thanks for looking at it! 1) The regular expression needs to be changed in generate-wait_event_types.pl. I have

Re: WIP: new system catalog pg_wait_event

2023-08-16 Thread Drouvot, Bertrand
Hi, On 8/16/23 2:08 PM, Michael Paquier wrote: On Wed, Aug 16, 2023 at 01:43:35PM +0200, Drouvot, Bertrand wrote: Yeah, agree, done that way in v6 (also added a test in 001_worker_spi.pl to ensure that "worker_spi_main" is reported in pg_wait_event). -type

Fix an entry in wait_event_names.txt

2023-08-16 Thread Drouvot, Bertrand
Hi hackers, While working on [1] it has been noticed by Masahiro-san that the description field in the new pg_wait_event view contains 2 blanks for one row. It turns out that it comes from wait_event_names.txt (added in fa88928). Attached a tiny patch to fix this entry in wait_event_names.txt

Re: WIP: new system catalog pg_wait_event

2023-08-16 Thread Drouvot, Bertrand
Hi, On 8/16/23 8:22 AM, Michael Paquier wrote: On Wed, Aug 16, 2023 at 07:04:53AM +0200, Drouvot, Bertrand wrote: I'd prefer the singular form. There is a lot of places where it's already used (pg_database, pg_user, pg_namespace...to name a few) and it looks like that using the plural form

Re: WIP: new system catalog pg_wait_event

2023-08-15 Thread Drouvot, Bertrand
Hi, On 8/14/23 6:37 AM, Michael Paquier wrote: On Thu, Aug 10, 2023 at 08:09:34PM +0200, Drouvot, Bertrand wrote: Agree that's worth it given the fact that iterating one more time is not that costly here. I have reviewed v4, and finished by putting my hands on it to see what I could do

Re: WIP: new system catalog pg_wait_event

2023-08-10 Thread Drouvot, Bertrand
Hi, On 8/9/23 9:56 AM, Michael Paquier wrote: On Tue, Aug 08, 2023 at 10:16:37AM +0200, Drouvot, Bertrand wrote: Please find attached v3 adding the wait event types. +-- There will surely be at least 9 wait event types, 240 wait events and at +-- least 27 related to WAL +select count

Re: WIP: new system catalog pg_wait_event

2023-08-08 Thread Drouvot, Bertrand
Hi, On 8/8/23 7:37 AM, Drouvot, Bertrand wrote: Hi, On 8/8/23 5:05 AM, Michael Paquier wrote: On Tue, Aug 08, 2023 at 11:53:32AM +0900, Kyotaro Horiguchi wrote: As I mentioned in another thread, I'm uncertain about our stance on the class id of the wait event. If a class acts as a namespace

Re: Synchronizing slots from primary to standby

2023-08-07 Thread Drouvot, Bertrand
Hi, On 8/8/23 7:01 AM, shveta malik wrote: On Mon, Aug 7, 2023 at 3:17 PM Drouvot, Bertrand wrote: Hi, On 8/4/23 1:32 PM, shveta malik wrote: On Fri, Aug 4, 2023 at 2:44 PM Drouvot, Bertrand wrote: On 7/28/23 4:39 PM, Bharath Rupireddy wrote: Agreed. That is why in v10,v11 patches

Re: WIP: new system catalog pg_wait_event

2023-08-07 Thread Drouvot, Bertrand
Hi, On 8/8/23 5:05 AM, Michael Paquier wrote: On Tue, Aug 08, 2023 at 11:53:32AM +0900, Kyotaro Horiguchi wrote: As I mentioned in another thread, I'm uncertain about our stance on the class id of the wait event. If a class acts as a namespace, we should include it in the view. Otherwise, if

Re: Fix badly generated entries in wait_event_names.txt

2023-08-07 Thread Drouvot, Bertrand
Hi, On 8/8/23 1:25 AM, Michael Paquier wrote: On Mon, Aug 07, 2023 at 10:34:43AM +0200, Drouvot, Bertrand wrote: fa88928470 introduced src/backend/utils/activity/wait_event_names.txt that has been auto-generated. The auto-generation had parsing errors leading to bad entries

Re: WIP: new system catalog pg_wait_event

2023-08-07 Thread Drouvot, Bertrand
Hi, On 8/7/23 10:23 AM, Drouvot, Bertrand wrote: Hi, On 8/4/23 5:08 PM, Tom Lane wrote: "Drouvot, Bertrand" writes: Now that fa88928470 generates automatically code and documentation related to wait events, why not exposing the wait events description through a system catalog rel

Re: Synchronizing slots from primary to standby

2023-08-07 Thread Drouvot, Bertrand
Hi, On 8/4/23 1:32 PM, shveta malik wrote: On Fri, Aug 4, 2023 at 2:44 PM Drouvot, Bertrand wrote: On 7/28/23 4:39 PM, Bharath Rupireddy wrote: Sorry to be late, but I gave a second thought and I wonder if we really need this design. (i.e start a logical replication background worker

Fix badly generated entries in wait_event_names.txt

2023-08-07 Thread Drouvot, Bertrand
Hi hackers, fa88928470 introduced src/backend/utils/activity/wait_event_names.txt that has been auto-generated. The auto-generation had parsing errors leading to bad entries in wait_event_names.txt (when the same "WAIT_EVENT" name can be found as a substring of another one, then descriptions

Re: WIP: new system catalog pg_wait_event

2023-08-07 Thread Drouvot, Bertrand
Hi, On 8/4/23 5:08 PM, Tom Lane wrote: "Drouvot, Bertrand" writes: Now that fa88928470 generates automatically code and documentation related to wait events, why not exposing the wait events description through a system catalog relation? I think you'd be better off making this a

Re: "duplicated" wait events

2023-08-07 Thread Drouvot, Bertrand
Hi, On 8/7/23 7:46 AM, Kyotaro Horiguchi wrote: At Fri, 4 Aug 2023 17:07:51 +0200, "Drouvot, Bertrand" wrote in SynRep currently appears in "IPC" and "LWLock" (see [2]) WALWrite currently appears in "IO" and "LWLock" (see [2]) I think

"duplicated" wait events

2023-08-04 Thread Drouvot, Bertrand
Hi hackers, while working on the new system catalog pg_wait_event (see [1]) I noticed that some wait events are currently "duplicated": postgres=# select wait_event_name,count(*) from pg_wait_event group by wait_event_name having count(*) > 1; wait_event_name | count

WIP: new system catalog pg_wait_event

2023-08-04 Thread Drouvot, Bertrand
Hi hackers, Now that fa88928470 generates automatically code and documentation related to wait events, why not exposing the wait events description through a system catalog relation? (the idea has been proposed on twitter by Yves Colin [1]). I think that could be useful to: - join this new

Re: Split index and table statistics into different types of stats

2023-08-04 Thread Drouvot, Bertrand
Hi, On 7/10/23 11:14 AM, Daniel Gustafsson wrote: On 4 Apr 2023, at 12:04, Drouvot, Bertrand wrote: On 4/3/23 11:47 PM, Gregory Stark (as CFM) wrote: On Thu, 16 Mar 2023 at 05:25, Drouvot, Bertrand wrote: My plan was to get [1] done before resuming working on the "Split index and

Re: Synchronizing slots from primary to standby

2023-08-04 Thread Drouvot, Bertrand
Hi, On 7/28/23 4:39 PM, Bharath Rupireddy wrote: On Mon, Jul 24, 2023 at 9:00 AM Amit Kapila wrote: 2. All candidate standbys will start one slot sync worker per logical slot which might not be scalable. Yeah, that doesn't sound like a good idea but IIRC, the proposed patch is using one

Re: Synchronizing slots from primary to standby

2023-08-04 Thread Drouvot, Bertrand
Hi, On 7/24/23 4:32 AM, Bharath Rupireddy wrote: On Fri, Jul 21, 2023 at 5:16 PM shveta malik wrote: Here are my thoughts about this feature: Thanks for looking at it! Important considerations: 1. Does this design guarantee the row versions required by subscribers aren't removed on

Re: Autogenerate some wait events code and documentation

2023-07-10 Thread Drouvot, Bertrand
On 7/11/23 12:52 AM, Michael Paquier wrote: On Mon, Jul 10, 2023 at 09:11:36AM +0200, Alvaro Herrera wrote: I don't like this bit, because it means the .txt file is now ungreppable as source of the enum name. Things become mysterious and people have to track down the event name by reading

Re: Autogenerate some wait events code and documentation

2023-07-09 Thread Drouvot, Bertrand
On 7/10/23 7:20 AM, Michael Paquier wrote: On Mon, Jul 10, 2023 at 07:05:30AM +0200, Drouvot, Bertrand wrote: Yeah there is one in generate-wait_event_types.pl. I was wondering to add one in wait_event_names.txt too (as this is the place where no wait events would be added if any). Hmm

Re: Autogenerate some wait events code and documentation

2023-07-09 Thread Drouvot, Bertrand
Hi, On 7/9/23 9:36 AM, Michael Paquier wrote: On Sun, Jul 09, 2023 at 09:15:34AM +0200, Drouvot, Bertrand wrote: I also noticed that you now provide the culprit line in case of parsing failure (thanks for that). Yes, that's mentioned in the commit message I quickly wrote in 0002

Re: Autogenerate some wait events code and documentation

2023-07-09 Thread Drouvot, Bertrand
Hi, On 7/9/23 6:32 AM, Michael Paquier wrote: On Fri, Jul 07, 2023 at 01:49:24PM +0900, Michael Paquier wrote: Hmm. If we go down this road I would make the choice of simplicity and remove entirely a column, then, generating the snakecase from the camelcase or vice-versa (say like a $string

Re: Autogenerate some wait events code and documentation

2023-07-04 Thread Drouvot, Bertrand
Hi, On 7/3/23 9:11 AM, Michael Paquier wrote: On Mon, Jul 03, 2023 at 03:57:42PM +0900, Michael Paquier wrote: Thanks for looking at it and having fixed the issues that were present in v10. I think that we should add some options to the perl script to be more selective with the files

Re: ProcessStartupPacket(): database_name and user_name truncation

2023-07-04 Thread Drouvot, Bertrand
Hi, On 7/3/23 10:34 PM, Nathan Bossart wrote: On Sat, Jul 01, 2023 at 04:02:06PM +0200, Drouvot, Bertrand wrote: Please find V2 attached where it's failing as soon as the database name or user name are detected as overlength. Thanks, Bertrand. I chickened out and ended up committing v1

Re: ProcessStartupPacket(): database_name and user_name truncation

2023-07-01 Thread Drouvot, Bertrand
Hi, On 6/30/23 7:32 PM, Drouvot, Bertrand wrote: Hi, On 6/30/23 5:54 PM, Tom Lane wrote: Nathan Bossart writes: After taking another look at this, I wonder if it'd be better to fail as soon as we see the database or user name is too long instead of lugging them around when authentication

Re: ProcessStartupPacket(): database_name and user_name truncation

2023-06-30 Thread Drouvot, Bertrand
Hi, On 6/30/23 5:54 PM, Tom Lane wrote: Nathan Bossart writes: After taking another look at this, I wonder if it'd be better to fail as soon as we see the database or user name is too long instead of lugging them around when authentication is destined to fail. If we're agreed that we aren't

Re: Synchronizing slots from primary to standby

2023-06-30 Thread Drouvot, Bertrand
Hi Kuroda-san, On 6/29/23 12:22 PM, Hayato Kuroda (Fujitsu) wrote: Dear Drouvot, Hi, I'm also interested in the feature. Followings are my high-level comments. I did not mention some detailed notations because this patch is not at the stage. And very sorry that I could not follow all of this

Re: Synchronizing slots from primary to standby

2023-06-29 Thread Drouvot, Bertrand
Hi, On 6/29/23 12:36 PM, Amit Kapila wrote: On Wed, Jun 28, 2023 at 12:19 PM Drouvot, Bertrand wrote: Yeah, I think once the slot is dropped we just have to wait for the slot to be re-created on the standby according to the new synchronize_slot_names GUC. Assuming the initial slot "cre

Re: Synchronizing slots from primary to standby

2023-06-28 Thread Drouvot, Bertrand
Hi, On 6/26/23 12:34 PM, Amit Kapila wrote: On Mon, Jun 26, 2023 at 11:15 AM Drouvot, Bertrand wrote: On 6/20/23 12:22 PM, Amit Kapila wrote: On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand wrote: In such a case (slot valid on the primary but invalidated on the standby) then I think

Re: Synchronizing slots from primary to standby

2023-06-25 Thread Drouvot, Bertrand
Hi, On 6/20/23 12:22 PM, Amit Kapila wrote: On Mon, Jun 19, 2023 at 9:56 PM Drouvot, Bertrand wrote: In such a case (slot valid on the primary but invalidated on the standby) then I think we could drop and recreate the invalidated slot on the standby. Will it be safe? Because after

Re: ProcessStartupPacket(): database_name and user_name truncation

2023-06-22 Thread Drouvot, Bertrand
Hi, On 6/22/23 1:37 AM, Michael Paquier wrote: On Wed, Jun 21, 2023 at 12:55:15PM -0700, Nathan Bossart wrote: LGTM. I think this can wait for v17 since the current behavior has been around since 2001 and AFAIK this is the first report. While it's arguably a bug fix, the patch also breaks

Re: ProcessStartupPacket(): database_name and user_name truncation

2023-06-21 Thread Drouvot, Bertrand
On 6/21/23 4:22 PM, Drouvot, Bertrand wrote: Hi, On 6/21/23 3:43 PM, Tom Lane wrote: Kyotaro Horiguchi writes: At Wed, 21 Jun 2023 09:43:50 +0200, "Drouvot, Bertrand" wrote in Trying to connect with the 64 bytes name: $ psql -d psql: error:

Re: ProcessStartupPacket(): database_name and user_name truncation

2023-06-21 Thread Drouvot, Bertrand
Hi, On 6/21/23 3:43 PM, Tom Lane wrote: Kyotaro Horiguchi writes: At Wed, 21 Jun 2023 09:43:50 +0200, "Drouvot, Bertrand" wrote in Trying to connect with the 64 bytes name: $ psql -d psql: error: connection to server on socket "/tmp/.s.PGSQ

ProcessStartupPacket(): database_name and user_name truncation

2023-06-21 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch to truncate (in ProcessStartupPacket()) the port->database_name and port->user_name in such a way to not break multibyte character boundary. Indeed, currently, one could create a database that way: postgres=# create database

Re: Synchronizing slots from primary to standby

2023-06-19 Thread Drouvot, Bertrand
Hi, On 6/19/23 12:03 PM, Amit Kapila wrote: On Mon, Jun 19, 2023 at 11:34 AM Drouvot, Bertrand wrote: Also I think we need to handle the case of invalidated replication slot(s): should we drop/recreate it/them? (as the main goal is to have sync slot(s) on the standby). Do you intend

Re: Synchronizing slots from primary to standby

2023-06-19 Thread Drouvot, Bertrand
Hi, On 6/16/23 11:56 AM, Amit Kapila wrote: On Mon, Apr 17, 2023 at 7:37 PM Drouvot, Bertrand wrote: Please find attached V5 (a rebase of V4 posted up-thread). In addition to the "rebasing" work, the TAP test adds a test about conflict handling (logical slot invalidatio

Re: Support to define custom wait events for extensions

2023-06-15 Thread Drouvot, Bertrand
Hi, On 6/15/23 10:00 AM, Michael Paquier wrote: On Thu, Jun 15, 2023 at 03:06:01PM +0900, Masahiro Ikeda wrote: Currently, only one PG_WAIT_EXTENSION event can be used as a wait event for extensions. Therefore, in environments with multiple extensions are installed, it could take time to

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-06-09 Thread Drouvot, Bertrand
Hi, On 6/9/23 11:20 AM, Masahiro Ikeda wrote: Hi, On 2023-06-09 13:26, Drouvot, Bertrand wrote: Hi, On 6/9/23 1:15 AM, Michael Paquier wrote: On Thu, Jun 08, 2023 at 10:57:55AM +0900, Masahiro Ikeda wrote: (Excuse me for cutting in, and this is not directly related to the thread.) +1. I'm

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-06-08 Thread Drouvot, Bertrand
Hi, On 6/9/23 1:15 AM, Michael Paquier wrote: On Thu, Jun 08, 2023 at 10:57:55AM +0900, Masahiro Ikeda wrote: (Excuse me for cutting in, and this is not directly related to the thread.) +1. I'm interested in the feature. This is just a example and it probable be useful for other users. IMO,

Re: Let's make PostgreSQL multi-threaded

2023-06-07 Thread Drouvot, Bertrand
Hi, On 6/8/23 12:37 AM, Jeremy Schneider wrote: On 6/7/23 2:39 PM, Thomas Kellerer wrote: Tomas Vondra schrieb am 07.06.2023 um 21:20: I did google search for "oracle threaded_execution" and browsed a bit; didn't see anything that seems earth shattering so far. FWIW, I recall Karl Arao's

Re: is pg_log_standby_snapshot() really needed?

2023-06-07 Thread Drouvot, Bertrand
Hi, On 6/7/23 8:50 PM, Jaime Casanova wrote: On Wed, Jun 7, 2023 at 5:19 AM Drouvot, Bertrand wrote: Hi, On 6/7/23 7:32 AM, Jaime Casanova wrote: So, I wonder if that function is really needed because as I said I solved it with already existing functionality. Or if it is really needed

Re: is pg_log_standby_snapshot() really needed?

2023-06-07 Thread Drouvot, Bertrand
Hi, On 6/7/23 7:32 AM, Jaime Casanova wrote: Hi, I'm testing the ability to have a logical replica subscribed from a standby. Of course, I'm doing this in a laboratory with no activity so everything get stuck after creating the subscription (the main slot). This is clearly because every time

Re: Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2023-05-31 Thread Drouvot, Bertrand
Hi, On 5/30/23 12:34 PM, Drouvot, Bertrand wrote: Hi hackers, Please find attached a patch proposal to $SUBJECT. Indeed, we have seen occurrences in [1] that some slots were not invalidated (while we expected vacuum to remove dead rows leading to slots invalidation on the standby). Though we

Re: BF animal dikkop reported a failure in 035_standby_logical_decoding

2023-05-30 Thread Drouvot, Bertrand
Hi, On 5/30/23 5:24 PM, Andres Freund wrote: Hi, On 2023-05-29 14:31:24 +0200, Drouvot, Bertrand wrote: On 5/29/23 1:03 PM, Tom Lane wrote: but I wouldn't be surprised if something in the logical replication mechanism itself could be running a transaction at the wrong instant. Some

Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2023-05-30 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch proposal to $SUBJECT. Indeed, we have seen occurrences in [1] that some slots were not invalidated (while we expected vacuum to remove dead rows leading to slots invalidation on the standby). Though we don't have strong evidences that this was due to

Re: BF animal dikkop reported a failure in 035_standby_logical_decoding

2023-05-29 Thread Drouvot, Bertrand
Hi, On 5/29/23 1:03 PM, Tom Lane wrote: "Drouvot, Bertrand" writes: On 5/26/23 9:27 AM, Yu Shi (Fujitsu) wrote: Is it possible that the vacuum command didn't remove tuples and then the conflict was not triggered? The flush_wal table added by Andres should guarantee that the WAL

Re: BF animal dikkop reported a failure in 035_standby_logical_decoding

2023-05-29 Thread Drouvot, Bertrand
Hi, On 5/26/23 9:27 AM, Yu Shi (Fujitsu) wrote: Hi hackers, I saw a buildfarm failure on "dikkop"[1]. It failed in 035_standby_logical_decoding.pl, because the slots row_removal_inactiveslot and row_removal_activeslot are not invalidated after vacuum. Thanks for sharing!

Re: Cleaning up nbtree after logical decoding on standby work

2023-05-29 Thread Drouvot, Bertrand
Hi, On 5/26/23 7:28 PM, Peter Geoghegan wrote: On Fri, May 26, 2023 at 1:56 AM Alvaro Herrera wrote: I suppose you're not thinking of applying this to current master, but instead just leave it for when pg17 opens, right? I mean, clearly it seems far too invasive to put it in after beta1. I

Re: pgsql: TAP test for logical decoding on standby

2023-05-24 Thread Drouvot, Bertrand
Hi, On 5/23/23 5:15 PM, Robert Haas wrote: On Sat, Apr 8, 2023 at 5:26 AM Andres Freund wrote: TAP test for logical decoding on standby Small nitpicks: 1. The test names generated by check_slots_conflicting_status() start with a capital letter, while most other test names start with a

Re: PG 16 draft release notes ready

2023-05-20 Thread Drouvot, Bertrand
Hi, On 5/19/23 2:29 PM, Bruce Momjian wrote: On Fri, May 19, 2023 at 09:49:18AM +0200, Drouvot, Bertrand wrote: Thanks! " This adds the function pg_log_standby_snapshot(). TEXT?: " My proposal: This adds the function pg_log_standby_snapshot() to log details of the current snaps

Re: PG 16 draft release notes ready

2023-05-19 Thread Drouvot, Bertrand
Hi, On 5/18/23 10:49 PM, Bruce Momjian wrote: I have completed the first draft of the PG 16 release notes. You can see the output here: https://momjian.us/pgsql_docs/release-16.html I will adjust it to the feedback I receive; that URL will quickly show all updates. Thanks! "

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-19 Thread Drouvot, Bertrand
Hi, On 5/19/23 12:36 AM, Michael Paquier wrote: On Thu, May 18, 2023 at 12:28:20PM -0400, Robert Haas wrote: I mean, I agree that it would probably be hard to measure any real performance difference. But I'm not sure that's a good reason to add cycles to a path where we don't really need to.

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-17 Thread Drouvot, Bertrand
Hi, On 5/16/23 8:16 AM, Michael Paquier wrote: On Mon, May 15, 2023 at 10:07:04AM +0200, Drouvot, Bertrand wrote: This is preliminary work to autogenerate some wait events code and documentation done in [1]. The patch introduces 2 new "wait events" (WAIT_EVENT

Re: walsender performance regression due to logical decoding on standby changes

2023-05-16 Thread Drouvot, Bertrand
Hi, On 5/15/23 4:39 PM, Bharath Rupireddy wrote: On Mon, May 15, 2023 at 6:14 PM Masahiko Sawada wrote: On Mon, May 15, 2023 at 1:49 PM Thomas Munro wrote: Do we need to add an open item for this issue in https://wiki.postgresql.org/wiki/PostgreSQL_16_Open_Items? If yes, can anyone in

Re: Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-16 Thread Drouvot, Bertrand
Hi, On 5/16/23 7:14 AM, Michael Paquier wrote: On Mon, May 15, 2023 at 06:01:02PM -0700, Andres Freund wrote: On 2023-05-16 09:38:54 +0900, Michael Paquier wrote: On Mon, May 15, 2023 at 05:17:16PM -0700, Andres Freund wrote: These are the two things refactored in the patch, explaining the

Re: walsender performance regression due to logical decoding on standby changes

2023-05-15 Thread Drouvot, Bertrand
Hi, On 5/15/23 4:19 AM, Zhijie Hou (Fujitsu) wrote: On Friday, May 12, 2023 7:58 PM Bharath Rupireddy wrote: On Wed, May 10, 2023 at 3:23 PM Drouvot, Bertrand That's not the case with the attached v2 patch. Please have a look. Thanks for V2! It does look good to me and I like the fact

Re: Autogenerate some wait events code and documentation

2023-05-15 Thread Drouvot, Bertrand
Hi, On 5/6/23 4:23 AM, Michael Paquier wrote: On Thu, May 04, 2023 at 08:39:49AM +0200, Drouvot, Bertrand wrote: On 5/1/23 1:59 AM, Michael Paquier wrote: I'm not sure I like it. First, it does break the "Note" ordering as compare to the current documentation. That's not a big d

Introduce WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN

2023-05-15 Thread Drouvot, Bertrand
Hi hackers, Please find attached a patch to $SUBJECT. This is preliminary work to autogenerate some wait events code and documentation done in [1]. The patch introduces 2 new "wait events" (WAIT_EVENT_EXTENSION and WAIT_EVENT_BUFFER_PIN) and their associated functions

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-11 Thread Drouvot, Bertrand
On 5/10/23 12:41 PM, Amit Kapila wrote: On Tue, May 9, 2023 at 12:44 PM Drouvot, Bertrand Pushed this yesterday. Thanks! -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: walsender performance regression due to logical decoding on standby changes

2023-05-10 Thread Drouvot, Bertrand
Hi, On 5/10/23 8:36 AM, Bharath Rupireddy wrote: On Wed, May 10, 2023 at 12:33 AM Andres Freund wrote: Unfortunately I have found the following commit to have caused a performance regression: commit e101dfac3a53c20bfbf1ca85d30a368c2954facf The problem is that, on a standby, after the

Re: walsender performance regression due to logical decoding on standby changes

2023-05-10 Thread Drouvot, Bertrand
Hi, On 5/9/23 11:00 PM, Andres Freund wrote: Hi, On 2023-05-09 13:38:24 -0700, Jeff Davis wrote: On Tue, 2023-05-09 at 12:02 -0700, Andres Freund wrote: I don't think the approach of not having any sort of "registry" of whether anybody is waiting for the replay position to be updated is

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-09 Thread Drouvot, Bertrand
Hi, On 5/9/23 8:02 AM, Amit Kapila wrote: On Mon, May 8, 2023 at 1:45 PM Drouvot, Bertrand wrote: Why not initialize the cascading standby node just before the standby promotion test: "Test standby promotion and logical decoding behavior after the standby gets promoted."? T

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-08 Thread Drouvot, Bertrand
Hi, On 5/8/23 4:42 AM, Amit Kapila wrote: On Sat, May 6, 2023 at 9:33 PM Drouvot, Bertrand wrote: On 5/6/23 3:28 PM, Amit Kapila wrote: On Sat, May 6, 2023 at 1:52 PM Drouvot, Bertrand wrote: Next steps: = 1. The first thing is we should verify this theory by adding some LOG

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-06 Thread Drouvot, Bertrand
Hi, On 5/6/23 3:28 PM, Amit Kapila wrote: On Sat, May 6, 2023 at 1:52 PM Drouvot, Bertrand wrote: There is 2 runs with this extra info in place: A successful one: https://cirrus-ci.com/task/6528745436086272 A failed one: https://cirrus-ci.com/task/4558139312308224 Thanks, I think I got

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-06 Thread Drouvot, Bertrand
Hi, On 5/6/23 4:10 AM, Amit Kapila wrote: On Fri, May 5, 2023 at 7:53 PM Drouvot, Bertrand wrote: On 5/5/23 2:28 PM, Amit Kapila wrote: On Fri, May 5, 2023 at 5:36 PM Drouvot, Bertrand So, even on a successful test, we can see that the WAL file we expect to be removed on the standby has

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-05 Thread Drouvot, Bertrand
On 5/5/23 2:28 PM, Amit Kapila wrote: On Fri, May 5, 2023 at 5:36 PM Drouvot, Bertrand wrote: It seems due to some reason the current wal file is not switched due to some reason. Oh wait, here is a NON failing one: https://cirrus-ci.com/task/5086849685782528 (I modified the .cirrus.yml

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-05 Thread Drouvot, Bertrand
On 5/5/23 12:58 PM, Amit Kapila wrote: On Fri, May 5, 2023 at 4:02 PM Drouvot, Bertrand wrote: How did you concluded that 00010003 is the file the test is expecting to be removed? because I added a note in the test that way: " @@ -535,6 +539,7 @@ $node_st

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-05 Thread Drouvot, Bertrand
On 5/5/23 11:29 AM, Amit Kapila wrote: On Fri, May 5, 2023 at 1:16 PM Drouvot, Bertrand wrote: After multiple attempts, I got one failing one. Issue is that we expect this file to be removed: [07:24:27.261](0.899s) #WAL file is /Users/admin/pgsql/build/testrun/recovery

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-05 Thread Drouvot, Bertrand
On 5/5/23 9:04 AM, Amit Kapila wrote: On Fri, May 5, 2023 at 11:08 AM Drouvot, Bertrand wrote: On 5/4/23 6:43 AM, Amit Kapila wrote: On Thu, May 4, 2023 at 8:37 AM vignesh C wrote: Thanks for posting the updated patch, I had run this test in a loop of 100 times to verify

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-05 Thread Drouvot, Bertrand
Hi, On 5/5/23 9:11 AM, vignesh C wrote: On Fri, 5 May 2023 at 12:34, Amit Kapila wrote: On Fri, May 5, 2023 at 11:08 AM Drouvot, Bertrand wrote: On 5/4/23 6:43 AM, Amit Kapila wrote: On Thu, May 4, 2023 at 8:37 AM vignesh C wrote: Thanks for posting the updated patch, I had run

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-04 Thread Drouvot, Bertrand
Hi, On 5/4/23 6:43 AM, Amit Kapila wrote: On Thu, May 4, 2023 at 8:37 AM vignesh C wrote: Thanks for posting the updated patch, I had run this test in a loop of 100 times to verify that there was no failure because of race conditions. The 100 times execution passed successfully. One

Re: Autogenerate some wait events code and documentation

2023-05-04 Thread Drouvot, Bertrand
Hi, On 5/2/23 4:50 AM, Thomas Munro wrote: [patch] This is not a review of the perl/make/meson glue/details, but I just wanted to say thanks for working on this Bertrand & Michael, at a quick glance that .txt file looks like it's going to be a lot more fun to maintain! Thanks Thomas! Yeah

Re: Autogenerate some wait events code and documentation

2023-05-04 Thread Drouvot, Bertrand
Hi, On 5/1/23 1:59 AM, Michael Paquier wrote: On Fri, Apr 28, 2023 at 02:29:13PM +0200, Drouvot, Bertrand wrote: On 4/27/23 8:13 AM, Michael Paquier wrote: But do we need to merge more data than necessary? We could do things in the simplest fashion possible while making the docs and code

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-03 Thread Drouvot, Bertrand
Hi, On 5/4/23 6:43 AM, Amit Kapila wrote: On Thu, May 4, 2023 at 8:37 AM vignesh C wrote: Thanks for posting the updated patch, I had run this test in a loop of 100 times to verify that there was no failure because of race conditions. The 100 times execution passed successfully. One

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-03 Thread Drouvot, Bertrand
Hi, On 5/3/23 12:29 PM, Amit Kapila wrote: On Tue, May 2, 2023 at 4:52 PM Drouvot, Bertrand wrote: As per your suggestion, changing the insert ordering (like in V8 attached) makes it now work on the failing environment too. I think it is better to use wait_for_replay_catchup() to wait

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-05-02 Thread Drouvot, Bertrand
Hi, On 5/2/23 8:28 AM, Amit Kapila wrote: On Fri, Apr 28, 2023 at 2:24 PM Drouvot, Bertrand wrote: I can see V7 failing on "Cirrus CI / macOS - Ventura - Meson" only (other machines are not complaining). It does fail on "invalidated logical slots do not lead to retaining W

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-28 Thread Drouvot, Bertrand
Hi, On 4/28/23 5:55 AM, Amit Kapila wrote: On Wed, Apr 26, 2023 at 7:53 PM Drouvot, Bertrand wrote: +# Get the restart_lsn from an invalidated slot +my $restart_lsn = $node_standby->safe_psql('postgres', + "SELECT restart_lsn from pg_replication_slots WHERE slot_name = 'vacuum_full_ac

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-27 Thread Drouvot, Bertrand
On 4/27/23 11:54 AM, Amit Kapila wrote: On Thu, Apr 27, 2023 at 1:05 PM Drouvot, Bertrand wrote: On 4/27/23 5:37 AM, Amit Kapila wrote: On Wed, Apr 26, 2023 at 4:41 PM Drouvot, Bertrand wrote: +When in recovery, the default value of target_lsn is $node->lsn('replay') +inst

Re: Fix a test case in 035_standby_logical_decoding.pl

2023-04-27 Thread Drouvot, Bertrand
Hi, On 4/27/23 11:53 AM, Amit Kapila wrote: On Thu, Apr 27, 2023 at 2:16 PM Drouvot, Bertrand wrote: On 4/27/23 10:11 AM, Yu Shi (Fujitsu) wrote: Hi hackers, In 035_standby_logical_decoding.pl, I think that the check in the following test case should be performed on the standby node

Re: Fix a test case in 035_standby_logical_decoding.pl

2023-04-27 Thread Drouvot, Bertrand
Hi, On 4/27/23 10:11 AM, Yu Shi (Fujitsu) wrote: Hi hackers, In 035_standby_logical_decoding.pl, I think that the check in the following test case should be performed on the standby node, instead of the primary node, as the slot is created on the standby node. Oh right, the current test is

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-27 Thread Drouvot, Bertrand
Hi, On 4/27/23 5:37 AM, Amit Kapila wrote: On Wed, Apr 26, 2023 at 4:41 PM Drouvot, Bertrand wrote: +When in recovery, the default value of target_lsn is $node->lsn('replay') +instead. This is needed when the publisher passed to wait_for_subscription_sync() +is a standby node. I th

Re: Autogenerate some wait events code and documentation

2023-04-26 Thread Drouvot, Bertrand
Hi, On 4/26/23 6:51 PM, Drouvot, Bertrand wrote: Hi, On 4/25/23 7:15 AM, Michael Paquier wrote: Will do, no problem at all. Please find attached V5 addressing the previous comments except the "ordering" one (need to look deeper at this). Regards, -- Bertrand Drouvot

Re: Autogenerate some wait events code and documentation

2023-04-26 Thread Drouvot, Bertrand
Hi, On 4/25/23 7:15 AM, Michael Paquier wrote: On Mon, Apr 24, 2023 at 09:03:53AM +0200, Drouvot, Bertrand wrote: Oh right, fixed. I may tweak a few things if I put my hands on it, but that looks pretty solid seen from here.. Glad to hear! ;-) I have spotted a few extra issues. One

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-26 Thread Drouvot, Bertrand
Hi, On 4/26/23 11:58 AM, Yu Shi (Fujitsu) wrote: On Mon, Apr 24, 2023 8:07 PM Drouvot, Bertrand wrote: I think that's because when replaying a checkpoint record, the startup process of standby only saves the information of the checkpoint, and we need to wait for the checkpointer to perform

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-26 Thread Drouvot, Bertrand
Hi, On 4/26/23 11:12 AM, vignesh C wrote: On Wed, 26 Apr 2023 at 13:45, Drouvot, Bertrand There was one typo in the commit message, subscribtion should be subscription, the rest of the changes looks good to me: Subject: [PATCH v5] Add subscribtion to the standby test

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-26 Thread Drouvot, Bertrand
Hi, On 4/26/23 12:27 PM, Alvaro Herrera wrote: diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index 6f7f4e5de4..819667d42a 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -2644,7 +2644,16

Re: Add two missing tests in 035_standby_logical_decoding.pl

2023-04-26 Thread Drouvot, Bertrand
Hi, On 4/26/23 6:06 AM, vignesh C wrote: On Tue, 25 Apr 2023 at 12:51, Drouvot, Bertrand wrote: Thanks for the updated patch. Few comments: Thanks for looking at it! 1) subscriber_stdout and subscriber_stderr are not required for this test case, we could remove it, I was able to remove

<    1   2   3   4   5   >