Re: Avoid orphaned objects dependencies, take 3

2024-06-21 Thread Bertrand Drouvot
Hi, On Wed, Jun 19, 2024 at 02:11:50PM +, Bertrand Drouvot wrote: > To sum up, I did not see any cases that did not lead to 1. or 2., so I think > it's safe to not add an extra lock for the RelationRelationId case. If, for > any > reason, there is still cases that are out

Re: Pluggable cumulative statistics

2024-06-20 Thread Bertrand Drouvot
n. Did not look yet. > - 0005 is an example of how to use them, with a TAP test providing > coverage. Did not look yet. As I said, I've in mind to do a more in depth review. I've noted the above while doing a quick read of the patches so thought it makes sense to share them now while at it. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Pluggable cumulative statistics

2024-06-20 Thread Bertrand Drouvot
ve done minus the in-core stats). This one would not be configurable and the extensions will not need to know about it. Would that remove the benefit from c) that you mentioned up-thread? [1]: https://www.postgresql.org/message-id/20220818195124.c7ipzf6c5v7vxymc%40awork3.anarazel.de Regards, -- Ber

Re: Avoid orphaned objects dependencies, take 3

2024-06-19 Thread Bertrand Drouvot
Hi, On Mon, Jun 17, 2024 at 05:57:05PM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Jun 17, 2024 at 12:24:46PM -0400, Robert Haas wrote: > > So to try to sum up here: I'm not sure I agree with this design. But I > > also feel like the design is not as clear and con

Re: Avoid orphaned objects dependencies, take 3

2024-06-17 Thread Bertrand Drouvot
Hi, On Mon, Jun 17, 2024 at 12:24:46PM -0400, Robert Haas wrote: > On Fri, Jun 14, 2024 at 3:54 AM Bertrand Drouvot > wrote: > > > Ah, right. So, I was assuming that, with either this version of your > > > patch or the earlier version, we'd end up locking the constraint &

Re: Allow logical failover slots to wait on synchronous replication

2024-06-17 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 10:00:46AM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Jun 10, 2024 at 09:25:10PM -0500, Nathan Bossart wrote: > > On Mon, Jun 10, 2024 at 03:51:05PM -0700, John H wrote: > > > The existing 'standby_slot_names' isn't great for users who ar

Re: Avoid orphaned objects dependencies, take 3

2024-06-17 Thread Bertrand Drouvot
Hi, On Thu, Jun 13, 2024 at 04:52:09PM +, Bertrand Drouvot wrote: > Hi, > > On Thu, Jun 13, 2024 at 10:49:34AM -0400, Robert Haas wrote: > > On Fri, Jun 7, 2024 at 4:41 AM Bertrand Drouvot > > wrote: > > > Do you still find the code hard to maintain w

Re: Avoid orphaned objects dependencies, take 3

2024-06-14 Thread Bertrand Drouvot
Hi, On Thu, Jun 13, 2024 at 02:27:45PM -0400, Robert Haas wrote: > On Thu, Jun 13, 2024 at 12:52 PM Bertrand Drouvot > wrote: > > > > table_open(childRelId, ...) would lock any "ALTER TABLE > > > > DROP CONSTRAINT" > > > > already. Not sure I

Re: Avoid orphaned objects dependencies, take 3

2024-06-13 Thread Bertrand Drouvot
Hi, On Thu, Jun 13, 2024 at 10:49:34AM -0400, Robert Haas wrote: > On Fri, Jun 7, 2024 at 4:41 AM Bertrand Drouvot > wrote: > > Do you still find the code hard to maintain with v9? > > I don't think it substantially changes my concerns as compared with > the ear

Re: Track the amount of time waiting due to cost_delay

2024-06-13 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 08:26:23AM +, Bertrand Drouvot wrote: > Hi, > > On Tue, Jun 11, 2024 at 04:07:05PM +0900, Masahiko Sawada wrote: > > > Thank you for the proposal and the patch. I understand the motivation > > of this patch. > > Thanks for looking

Re: relfilenode statistics

2024-06-12 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 03:35:23PM +0900, Kyotaro Horiguchi wrote: > At Mon, 10 Jun 2024 08:09:56 +0000, Bertrand Drouvot > wrote in > > > > My idea was to move all that is in pg_statio_all_tables to relfilenode stats > > and 1) add new stats to pg

Re: Track the amount of time waiting due to cost_delay

2024-06-12 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 01:13:48PM -0400, Robert Haas wrote: > On Tue, Jun 11, 2024 at 5:49 AM Bertrand Drouvot > wrote: > > As we can see the actual wait time is 30ms less than the intended wait time > > with > > this simple test. So I still think we should go

Re: Track the amount of time waiting due to cost_delay

2024-06-12 Thread Bertrand Drouvot
o the time waited). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-12 Thread Bertrand Drouvot
Hi, On Tue, Jun 11, 2024 at 11:40:36AM -0500, Nathan Bossart wrote: > On Tue, Jun 11, 2024 at 07:25:11AM +0000, Bertrand Drouvot wrote: > > So I think that in v2 we could: 1) measure the actual wait time instead, 2) > > count the number of times the vacuum slept. We could

Re: Allow logical failover slots to wait on synchronous replication

2024-06-11 Thread Bertrand Drouvot
e of the > replicas falls behind for whatever reason. A way to tie standby_slot_names > to synchronous replication instead does seem like it would be useful in > this case. FWIW, I have the same understanding and also think your proposal would be useful in this case. Regards, -- Bertrand Drouvot Postg

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
xcOM9Hlg%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 1345e99dcb..e4ba8de00

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
another dedicated view? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
sleading but I'll yield to majority opinion if people think it's not). Thoughts? [1]: https://www.postgresql.org/message-id/A0935130-7C4B-4094-B6E4-C7D5086D50EF%40amazon.com Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
tivity a using (pid) > > Maybe all progress views should just expose the > "beentry->st_activity_start_timestamp " > to let the user know when the current operation began. Yeah maybe, I think this is likely a separate discussion too, thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Track the amount of time waiting due to cost_delay

2024-06-11 Thread Bertrand Drouvot
On Mon, Jun 10, 2024 at 02:20:16PM -0500, Nathan Bossart wrote: > On Mon, Jun 10, 2024 at 05:48:22PM +0000, Bertrand Drouvot wrote: > > On Mon, Jun 10, 2024 at 10:36:42AM -0500, Nathan Bossart wrote: > >> I wonder if we should also > >> surface the effective cost limit

Re: Track the amount of time waiting due to cost_delay

2024-06-10 Thread Bertrand Drouvot
Hi, On Mon, Jun 10, 2024 at 10:36:42AM -0500, Nathan Bossart wrote: > On Mon, Jun 10, 2024 at 06:05:13AM +0000, Bertrand Drouvot wrote: > > During the last pgconf.dev I attended Robert´s presentation about > > autovacuum and > > it made me remember of an idea I had so

Re: relfilenode statistics

2024-06-10 Thread Bertrand Drouvot
user can still retrieve the stats from pg_statio_all_tables in such a way that it survives a rewrite, 3) provide dedicated APIs to retrieve relfilenode stats but only for the current relfilenode, 4) document this behavior. This is what the POC patch is doing for heap_blks_written (would need to do

Re: Proposal to add page headers to SLRU pages

2024-06-10 Thread Bertrand Drouvot
of pg_upgrade. I think there is something in t/002_pg_upgrade.pl (see src/bin/pg_upgrade/TESTING), that could be used to run automated tests using an old and a current version. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2024-06-10 Thread Bertrand Drouvot
Hi, On Mon, Jun 10, 2024 at 03:39:34PM +0900, Michael Paquier wrote: > On Mon, Jun 10, 2024 at 06:29:17AM +0000, Bertrand Drouvot wrote: > > Thanks for the report! I think it makes sense to add it to the list of known > > failures. > > > > One way to deal with those co

Re: Test slots invalidations in 035_standby_logical_decoding.pl only if dead rows are removed

2024-06-10 Thread Bertrand Drouvot
Hi Alexander, On Sat, Jun 08, 2024 at 07:00:00AM +0300, Alexander Lakhin wrote: > Hello Bertrand and Michael, > > 23.01.2024 11:07, Bertrand Drouvot wrote: > > On Tue, Jan 23, 2024 at 02:50:06PM +0900, Michael Paquier wrote: > > > > > Anyway, that's not the

Track the amount of time waiting due to cost_delay

2024-06-10 Thread Bertrand Drouvot
stop-doing-that/ Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 750dfc26cd6fcf5a5618c3fe35fc42d5b5c66f00 Mon Sep 17 00:00:00 2001 From: Bertrand Drouvot Date: Thu, 6 Jun 2024 1

Re: relfilenode statistics

2024-06-07 Thread Bertrand Drouvot
Hi, On Thu, Jun 06, 2024 at 08:17:36PM -0700, Andres Freund wrote: > Hi, > > On 2024-06-06 12:27:49 -0400, Robert Haas wrote: > > On Wed, Jun 5, 2024 at 1:52 AM Bertrand Drouvot > > wrote: > > > I think we should keep the stats in the relation during relfilenode

Re: relfilenode statistics

2024-06-07 Thread Bertrand Drouvot
Hi, On Thu, Jun 06, 2024 at 08:38:06PM -0700, Andres Freund wrote: > Hi, > > On 2024-06-03 11:11:46 +, Bertrand Drouvot wrote: > > The main argument is that we currently don’t have writes counters for > > relations. > > The reason is that we don’t have the relati

Re: Avoid orphaned objects dependencies, take 3

2024-06-07 Thread Bertrand Drouvot
Hi, On Thu, Jun 06, 2024 at 04:00:23PM -0400, Robert Haas wrote: > On Thu, Jun 6, 2024 at 1:56 AM Bertrand Drouvot > wrote: > > v9 is more invasive (as it changes code in much more places) than v8 but it > > is > > easier to follow (as it is now clear where the new loc

Re: How about using dirty snapshots to locate dependent objects?

2024-06-06 Thread Bertrand Drouvot
phaned dependencies cases. There is an ongoing thread (see [1]) to fix the orphaned dependencies issue. v9 attached in [1] fixes the case you describe here. [1]: https://www.postgresql.org/message-id/flat/ZiYjn0eVc7pxVY45%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: relfilenode statistics

2024-06-06 Thread Bertrand Drouvot
Hi, On Mon, Jun 03, 2024 at 11:11:46AM +, Bertrand Drouvot wrote: > Yeah, I’ll update the commit message in V2 with better explanations once I get > feedback on V1 (should we decide to move on with the relfilenode stats idea). > Please find attached v2, mandatory rebase due to c

Re: Avoid orphaned objects dependencies, take 3

2024-06-05 Thread Bertrand Drouvot
Hi, On Thu, May 23, 2024 at 02:10:54PM -0400, Robert Haas wrote: > On Thu, May 23, 2024 at 12:19 AM Bertrand Drouvot > wrote: > > The reason why we are using a dirty snapshot here is for the cases where we > > are > > recording a dependency on a referenced obj

Re: relfilenode statistics

2024-06-04 Thread Bertrand Drouvot
On Tue, Jun 04, 2024 at 09:26:27AM -0400, Robert Haas wrote: > On Mon, Jun 3, 2024 at 7:11 AM Bertrand Drouvot > wrote: > > We’d move the current buffer read and buffer hit counters from the relation > > stats > > to the relfilenode stats (while still being ab

Re: relfilenode statistics

2024-06-03 Thread Bertrand Drouvot
commit message. The body of the email where you post the patch > set can be fine, too. > Yeah, I’ll update the commit message in V2 with better explanations once I get feedback on V1 (should we decide to move on with the relfilenode stats idea). Regards, -- Bertrand Drouvot Pos

relfilenode statistics

2024-05-25 Thread Bertrand Drouvot
for the new heap_blks_written one ( should we agree on it). Looking forward to your comments, feedback. Regards, [1]: https://www.postgresql.org/message-id/20231113204439.r4lmys73tessqmak%40awork3.anarazel.de -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databa

Re: Avoid orphaned objects dependencies, take 3

2024-05-23 Thread Bertrand Drouvot
Hi, On Thu, May 23, 2024 at 02:10:54PM -0400, Robert Haas wrote: > On Thu, May 23, 2024 at 12:19 AM Bertrand Drouvot > wrote: > > The reason why we are using a dirty snapshot here is for the cases where we > > are > > recording a dependency on a referenced obj

Re: PostgreSQL 17 Beta 1 release announcement draft

2024-05-22 Thread Bertrand Drouvot
anks! Works for me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2024-05-22 Thread Bertrand Drouvot
Hi, On Wed, May 22, 2024 at 10:48:12AM -0400, Robert Haas wrote: > On Wed, May 22, 2024 at 6:21 AM Bertrand Drouvot > wrote: > > I started initially with [1] but it was focusing on function-schema only. > > Yeah, that's what I thought we would want to do. An

Re: Avoid orphaned objects dependencies, take 3

2024-05-22 Thread Bertrand Drouvot
Hi, On Tue, May 21, 2024 at 08:53:06AM -0400, Robert Haas wrote: > On Wed, May 15, 2024 at 6:31 AM Bertrand Drouvot > wrote: > > Please find attached v6 (only diff with v5 is moving the tests as suggested > > above). > > I don't immediately know what to think a

Re: Avoid orphaned objects dependencies, take 3

2024-05-21 Thread Bertrand Drouvot
thought about this name as it is close enough to the already existing "ERRCODE_DEPENDENT_OBJECTS_STILL_EXIST" but I'm open to suggestion too. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From 79809f8

Re: PostgreSQL 17 Beta 1 release announcement draft

2024-05-20 Thread Bertrand Drouvot
Hi, On Sun, May 19, 2024 at 05:10:10PM -0400, Jonathan S. Katz wrote: > On 5/16/24 1:15 AM, Bertrand Drouvot wrote: > > Hi, > > > > On Wed, May 15, 2024 at 09:45:35PM -0400, Jonathan S. Katz wrote: > > > Hi, > > > > > > Attached is a copy of the

Re: PostgreSQL 17 Beta 1 release announcement draft

2024-05-15 Thread Bertrand Drouvot
hat a session is waiting for (should it be active)"? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Avoid orphaned objects dependencies, take 3

2024-05-15 Thread Bertrand Drouvot
Hi, On Wed, May 15, 2024 at 08:31:43AM +, Bertrand Drouvot wrote: > Hi, > > On Wed, May 15, 2024 at 10:14:09AM +0900, Michael Paquier wrote: > > +++ > > b/src/test/modules/test_dependencies_locks/specs/test_dependencies_locks.spec > > > > > > This i

Re: Avoid orphaned objects dependencies, take 3

2024-05-15 Thread Bertrand Drouvot
Hi, On Tue, May 14, 2024 at 03:00:00PM +0300, Alexander Lakhin wrote: > Hi Bertrand, > > 09.05.2024 15:20, Bertrand Drouvot wrote: > > Oh I see, your test updates an existing dependency. v4 took care about > > brand new > > dependencies creation (recordMultipleDepende

Re: Avoid orphaned objects dependencies, take 3

2024-05-15 Thread Bertrand Drouvot
Hi, On Wed, May 15, 2024 at 10:14:09AM +0900, Michael Paquier wrote: > On Thu, May 09, 2024 at 12:20:51PM +0000, Bertrand Drouvot wrote: > > Oh I see, your test updates an existing dependency. v4 took care about > > brand new > > dependencies creation (recordMultipleDepen

Re: Log details for stats dropped more than once

2024-05-15 Thread Bertrand Drouvot
Hi, On Wed, May 15, 2024 at 02:47:29PM +0900, Michael Paquier wrote: > On Tue, May 14, 2024 at 10:07:14AM +0000, Bertrand Drouvot wrote: > > While resuming the work on refilenode stats (mentioned in [1] but did not > > share > > the patch yet), I realized that my curre

Log details for stats dropped more than once

2024-05-14 Thread Bertrand Drouvot
/ZbIdgTjR2QcFJ2mE%40ip-10-97-1-34.eu-west-3.compute.internal Looking forward to your feedback, Regards -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com >From f296d4092f5e87ed63a323db3f1bc9cfa807e2e8 Mon Sep 17 00:00:00 2001 From: Bertr

Re: Avoid orphaned objects dependencies, take 3

2024-05-09 Thread Bertrand Drouvot
dependency (which is done in another code path: changeDependencyFor()). Please find attached v5 that adds: - a call to the new depLockAndCheckObject() function in changeDependencyFor(). - a test when altering an existing dependency. With v5 applied, I don't see the issue anymore. Regards,

Re: First draft of PG 17 release notes

2024-05-08 Thread Bertrand Drouvot
t; Add system view pg_wait_events that reports wait event types (Michael > Paquier) Michael is the committer for 1e68e43d3f, the author is me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-05-08 Thread Bertrand Drouvot
57 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > On Friday, March 15, 2024 10:45 PM Bertrand Drouvot > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > On Thu, Mar 14, 2024 at 02:22:44AM +, Zhijie Hou

Re: Avoid orphaned objects dependencies, take 3

2024-04-25 Thread Bertrand Drouvot
Hi, On Thu, Apr 25, 2024 at 09:00:00AM +0300, Alexander Lakhin wrote: > Hi, > > 25.04.2024 08:00, Bertrand Drouvot wrote: > > > > > though concurrent create/drop operations can still produce > > > the "cache lookup failed" error, which is probably okay

Re: Avoid orphaned objects dependencies, take 3

2024-04-24 Thread Bertrand Drouvot
Hi, On Wed, Apr 24, 2024 at 03:00:00PM +0300, Alexander Lakhin wrote: > 24.04.2024 11:38, Bertrand Drouvot wrote: > > I gave more thought to v2 and the approach seems reasonable to me. > > Basically what > > it does is that in case the object is already dropped before we t

Re: Avoid orphaned objects dependencies, take 3

2024-04-24 Thread Bertrand Drouvot
Hi, On Tue, Apr 23, 2024 at 04:20:46PM +, Bertrand Drouvot wrote: > Please find attached v2 that should not produce the issue anymore (I launched > a > lot of attempts without any issues). v1 was not strong enough as it was not > always checking for the dependent object exist

Re: Avoid orphaned objects dependencies, take 3

2024-04-23 Thread Bertrand Drouvot
Hi, On Tue, Apr 23, 2024 at 04:59:09AM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Apr 22, 2024 at 03:00:00PM +0300, Alexander Lakhin wrote: > > 22.04.2024 13:52, Bertrand Drouvot wrote: > > > > > > That's weird, I just launched it several times with

Re: Avoid orphaned objects dependencies, take 3

2024-04-22 Thread Bertrand Drouvot
Hi, On Mon, Apr 22, 2024 at 03:00:00PM +0300, Alexander Lakhin wrote: > 22.04.2024 13:52, Bertrand Drouvot wrote: > > > > That's weird, I just launched it several times with the patch applied and > > I'm not > > able to see the seg fault (while I can see it constently

Re: Avoid orphaned objects dependencies, take 3

2024-04-22 Thread Bertrand Drouvot
Hi, On Mon, Apr 22, 2024 at 01:00:00PM +0300, Alexander Lakhin wrote: > Hi Bertrand, > > 22.04.2024 11:45, Bertrand Drouvot wrote: > > Hi, > > > > This new thread is a follow-up of [1] and [2]. > > > > Problem description: > > > > We ha

Re: Disallow changing slot's failover option in transaction block

2024-04-22 Thread Bertrand Drouvot
Hi, On Mon, Apr 22, 2024 at 11:32:14AM +0530, shveta malik wrote: > On Mon, Apr 22, 2024 at 5:57 AM Zhijie Hou (Fujitsu) > wrote: > > Attach the V3 patch which also addressed Shveta[1] and Bertrand[2]'s > > comments. Thanks! > Tested the patch, works well. Same here. R

Re: Patch to avoid orphaned dependencies

2024-04-22 Thread Bertrand Drouvot
cost (though not to zero). Thanks for the idea (and sorry for the delay replying to it)! I had a look at it and just created a new thread [1] based on your proposal. [1]: https://www.postgresql.org/message-id/flat/ZiYjn0eVc7pxVY45%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Ber

Avoid orphaned objects dependencies, take 3

2024-04-22 Thread Bertrand Drouvot
te that I'm not used to with this area of the code so that the patch might not take the option proposed by Tom the "right" way. Adding the patch to the July CF. Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databa

Re: Disallow changing slot's failover option in transaction block

2024-04-19 Thread Bertrand Drouvot
so I think it needs a bit of rewording (as here it sounds like the sub is already created), , something like? "Always ensure the slot property failover matches the counterpart parameter of the subscription and vice versa." Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS

Re: promotion related handling in pg_sync_replication_slots()

2024-04-18 Thread Bertrand Drouvot
aid that, the logic that is in place to take care of the corner cases described up-thread seems reasonable to me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-16 Thread Bertrand Drouvot
ase as well, SQL function will ultimately get error "cannot > synchronize replication slots concurrently", even though GUC is > enabled. Thus, I feel we should stick with samer error in all > scenarios. Okay, fine by me, let's forget about checking sync_replication_slots then. R

Re: Disallow changing slot's failover option in transaction block

2024-04-16 Thread Bertrand Drouvot
option as true cannot be executed inside a transaction block" Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-16 Thread Bertrand Drouvot
s() and emmit an error if sync_replication_slots is set to on? (The message could explicitly states that it's not possible to use the function if sync_replication_slots is set to on). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-16 Thread Bertrand Drouvot
Hi, On Tue, Apr 16, 2024 at 08:21:04AM +0530, Amit Kapila wrote: > On Mon, Apr 15, 2024 at 7:47 PM Bertrand Drouvot > wrote: > > > > On Mon, Apr 15, 2024 at 06:29:49PM +0530, Amit Kapila wrote: > > > On Mon, Apr 15, 2024 at 6:21 PM Bertrand Drouvot > > > w

Re: promotion related handling in pg_sync_replication_slots()

2024-04-15 Thread Bertrand Drouvot
Hi, On Mon, Apr 15, 2024 at 06:29:49PM +0530, Amit Kapila wrote: > On Mon, Apr 15, 2024 at 6:21 PM Bertrand Drouvot > wrote: > > > > On Mon, Apr 15, 2024 at 03:38:53PM +0530, shveta malik wrote: > > > Now both worker and SQL > > > function set 'syncin

Re: promotion related handling in pg_sync_replication_slots()

2024-04-15 Thread Bertrand Drouvot
th worker and SQL > function set 'syncing' when they start and reset it when they exit. It means that it's not possible anymore to trigger a manual sync if sync_replication_slots is on. Indeed that would trigger: postgres=# select pg_sync_replication_slots(); ERROR: cannot synchronize repl

Re: Synchronizing slots from primary to standby

2024-04-06 Thread Bertrand Drouvot
Hi, On Sat, Apr 06, 2024 at 10:13:00AM +0530, Amit Kapila wrote: > On Fri, Apr 5, 2024 at 8:05 PM Bertrand Drouvot > wrote: > > I think the new LSN can be visible only when the corresponding WAL is > written by XLogWrite(). I don't know what in XLogSetAsyncXactLSN() can &g

Re: Synchronizing slots from primary to standby

2024-04-05 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 02:35:42PM +, Bertrand Drouvot wrote: > I think that maybe as a first step we should move the "elog(DEBUG2," message > as > proposed above to help debugging (that could help to confirm the above > theory). If you agree and think that m

Re: Synchronizing slots from primary to standby

2024-04-05 Thread Bertrand Drouvot
age as proposed above to help debugging (that could help to confirm the above theory). If the theory is proven then I'm not sure we need the extra complexity of injection point here, maybe just relying on the slots created via SQL API could be enough. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-04-05 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 04:09:01PM +0530, shveta malik wrote: > On Fri, Apr 5, 2024 at 10:09 AM Bertrand Drouvot > wrote: > > > > What about something like? > > > > ereport(LOG, > > errmsg("synchronized confirmed_flush_lsn for

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-05 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 11:21:43AM +0530, Bharath Rupireddy wrote: > On Wed, Apr 3, 2024 at 9:57 PM Bertrand Drouvot > wrote: > Please find the attached v36 patch. Thanks! A few comments: 1 === + +The timeout is measured from the time since the slot h

Re: Synchronizing slots from primary to standby

2024-04-04 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 09:43:35AM +0530, shveta malik wrote: > On Fri, Apr 5, 2024 at 9:22 AM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Thu, Apr 04, 2024 at 05:31:45PM +0530, shveta malik wrote: > > > On Thu, Apr 4, 2024 at 2:

Re: Synchronizing slots from primary to standby

2024-04-04 Thread Bertrand Drouvot
) ", +remote_slot->name, + LSN_FORMAT_ARGS(slot->data.confirmed_flush), + LSN_FORMAT_ARGS(remote_slot->confirmed_lsn)); I don't think that the message is correct here. Unless I am mis

Re: Autogenerate some wait events code and documentation

2024-04-04 Thread Bertrand Drouvot
Hi, On Thu, Apr 04, 2024 at 07:14:47PM +0900, Michael Paquier wrote: > On Thu, Apr 04, 2024 at 09:28:36AM +0000, Bertrand Drouvot wrote: > > +# No "Backpatch" region here as code is generated automatically. > > > > What about "region here as has

Re: Autogenerate some wait events code and documentation

2024-04-04 Thread Bertrand Drouvot
Hi, On Thu, Apr 04, 2024 at 03:50:21PM +0900, Michael Paquier wrote: > On Tue, Mar 19, 2024 at 07:34:09AM +0000, Bertrand Drouvot wrote: > > I'm not sure as v2 used the "version >= 17" wording which I think would not > > need > > manual refresh each

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
Hi, On Wed, Apr 03, 2024 at 08:28:04PM +0530, Bharath Rupireddy wrote: > On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot > wrote: > > > > Just one comment on v32-0001: > > > > +# Synced slot on the standby must get its own inactive_since.

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
think we should just add some usleep() where appropriate and deny equality during the tests on inactive_since. Except for the above, v32-0001 LGTM. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
cation_slot_inactive_timeout/? CR9 === +# Start: Helper functions used for this test file +# End: Helper functions used for this test file I think that's the first TAP test with this comment. Not saying we should not but why did you feel the need to add those? [1]: https://www.postgresql.org/messa

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
=== + # Check that the captured time is sane + if (defined $reference_time) + { s/Check that the captured time is sane/Check that the inactive_since is sane/? Sorry if some of those comments could have been done while I did review v29-0001. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-04-02 Thread Bertrand Drouvot
Hi, On Tue, Apr 02, 2024 at 02:19:30PM +0530, Amit Kapila wrote: > On Tue, Apr 2, 2024 at 1:54 PM Bertrand Drouvot > wrote: > > What about adding a "wait" injection point in LogStandbySnapshot() to > > prevent > > checkpointer/bgwriter to log a standby snapshot

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-02 Thread Bertrand Drouvot
Hi, On Tue, Apr 02, 2024 at 12:41:35PM +0530, Bharath Rupireddy wrote: > On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot > wrote: > > > > > Or a simple solution is that the slotsync worker updates > > > inactive_since as it does for non-synced slots, and dis

Re: Synchronizing slots from primary to standby

2024-04-02 Thread Bertrand Drouvot
if (AmCheckpointerProcess() || AmBackgroundWriterProcess()) INJECTION_POINT("bgw-log-standby-snapshot"); And make use of it in the test, something like: $node_primary->safe_psql('postgres', "SELECT injection_points_attach('bgw-log-standby-s

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-02 Thread Bertrand Drouvot
r a simple solution is that the slotsync worker updates > inactive_since as it does for non-synced slots, and disables > timeout-based slot invalidation for synced slots. Yeah, I think the main question to help us decide is: do we want to invalidate "inactive" synced slots locally (in addition to synchronizing the invalidation from the primary)? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-04-01 Thread Bertrand Drouvot
Hi, On Tue, Apr 02, 2024 at 04:24:49AM +, Zhijie Hou (Fujitsu) wrote: > On Monday, April 1, 2024 9:28 PM Bertrand Drouvot > wrote: > > > > On Mon, Apr 01, 2024 at 05:04:53PM +0530, Amit Kapila wrote: > > > On Mon, Apr 1, 2024 at 2:51 PM Bertrand D

Re: Synchronizing slots from primary to standby

2024-04-01 Thread Bertrand Drouvot
Hi, On Mon, Apr 01, 2024 at 05:04:53PM +0530, Amit Kapila wrote: > On Mon, Apr 1, 2024 at 2:51 PM Bertrand Drouvot > wrote: > > Then there is no need to call WaitForStandbyConfirmation() as it could go > > until > > the RecoveryInProgress() in StandbySlotsHaveCaugh

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Bertrand Drouvot
Hi, On Sun, Mar 31, 2024 at 10:25:46AM +0530, Bharath Rupireddy wrote: > On Thu, Mar 28, 2024 at 3:13 PM Bertrand Drouvot > wrote: > > I think in this case it should always reflect the value from the primary (so > > that one can understand why it is invalidated). &

Re: Synchronizing slots from primary to standby

2024-04-01 Thread Bertrand Drouvot
alize() maintains a "small list" of "already serialized" snapshots. [1]: https://www.postgresql.org/message-id/ZgayTFIhLfzhpHci%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Bertrand Drouvot
the data, and if the user needs helps to make sense of that data, then give them that help separately. " [1]: https://www.postgresql.org/message-id/CAA4eK1JtKieWMivbswYg5FVVB5FugCftLvQKVsxh%3Dm_8nk04vw%40mail.gmail.com [2]: https://www.postgresql.org/message-id/CA%2BTgmob_Ta-t2ty8QrKHBGnNLrf4ZYcwhGHGFsuUoFrAEDw4sA%40mail.gmail.com Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Bertrand Drouvot
Hi, On Mon, Apr 01, 2024 at 09:04:43AM +0530, Amit Kapila wrote: > On Fri, Mar 29, 2024 at 6:17 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 29, 2024 at 03:03:01PM +0530, Amit Kapila wrote: > > > On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot > > > w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 03:03:01PM +0530, Amit Kapila wrote: > On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot > wrote: > > > > On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote: > > > > > > Commit message states: "why we can't just upd

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 02:35:22PM +0530, Amit Kapila wrote: > On Fri, Mar 29, 2024 at 1:08 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 29, 2024 at 07:23:11AM +, Zhijie Hou (Fujitsu) wrote: > > > On Friday, March 29, 2024 2:48 PM Bertrand Drouvot >

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 07:23:11AM +, Zhijie Hou (Fujitsu) wrote: > On Friday, March 29, 2024 2:48 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Fri, Mar 29, 2024 at 01:06:15AM +, Zhijie Hou (Fujitsu) wrote: > > > Attach a new vers

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
n this hidden slot only to advance to the max lsn of the synced slots I'm not sure that would be enough, just asking your thoughts on this (benefits would be to avoid calling pg_logical_replication_slot_advance() on each sync slots and during the sync cycles). Regards, -- Bertrand Drouvot Pos

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-29 Thread Bertrand Drouvot
t. Right, but for a very active slot it is likely that we call GetCurrentTimestamp() during almost each sync cycle. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-03-28 Thread Bertrand Drouvot
Hi, On Thu, Mar 28, 2024 at 05:05:35PM +0530, Amit Kapila wrote: > On Thu, Mar 28, 2024 at 3:34 PM Bertrand Drouvot > wrote: > > > > On Thu, Mar 28, 2024 at 04:38:19AM +, Zhijie Hou (Fujitsu) wrote: > > > > > To fix this, we could use the fast fo

Re: Synchronizing slots from primary to standby

2024-03-28 Thread Bertrand Drouvot
ndering about the performance impact (even in fast_forward mode), might be worth to keep an eye on it. Should we create a 17 open item [1]? [1]: https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-28 Thread Bertrand Drouvot
ePossiblyObsoleteSlot() is not called with such a reason, better to use an Assert here and in the caller too? CR11 === +++ b/src/test/recovery/t/050_invalidate_slots.pl why not using 019_replslot_limit.pl? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-27 Thread Bertrand Drouvot
Hi, On Wed, Mar 27, 2024 at 09:00:37PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 27, 2024 at 6:54 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote: > > > On Wed, Mar 27, 2024 at 3:

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-27 Thread Bertrand Drouvot
Hi, On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot > Please see the attached v28 patch. Thanks! 1 === sorry I missed it in the previous review if (!(RecoveryInProgress() && slo

  1   2   3   >