Re: Avoid orphaned objects dependencies, take 3

2024-05-09 Thread Bertrand Drouvot
dependency (which is done in another code path: changeDependencyFor()). Please find attached v5 that adds: - a call to the new depLockAndCheckObject() function in changeDependencyFor(). - a test when altering an existing dependency. With v5 applied, I don't see the issue anymore. Regards,

Re: First draft of PG 17 release notes

2024-05-08 Thread Bertrand Drouvot
t; Add system view pg_wait_events that reports wait event types (Michael > Paquier) Michael is the committer for 1e68e43d3f, the author is me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-05-08 Thread Bertrand Drouvot
57 AM Zhijie Hou (Fujitsu) > > > wrote: > > > > > > > > On Friday, March 15, 2024 10:45 PM Bertrand Drouvot > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > On Thu, Mar 14, 2024 at 02:22:44AM +, Zhijie Hou

Re: Avoid orphaned objects dependencies, take 3

2024-04-25 Thread Bertrand Drouvot
Hi, On Thu, Apr 25, 2024 at 09:00:00AM +0300, Alexander Lakhin wrote: > Hi, > > 25.04.2024 08:00, Bertrand Drouvot wrote: > > > > > though concurrent create/drop operations can still produce > > > the "cache lookup failed" error, which is probably okay

Re: Avoid orphaned objects dependencies, take 3

2024-04-24 Thread Bertrand Drouvot
Hi, On Wed, Apr 24, 2024 at 03:00:00PM +0300, Alexander Lakhin wrote: > 24.04.2024 11:38, Bertrand Drouvot wrote: > > I gave more thought to v2 and the approach seems reasonable to me. > > Basically what > > it does is that in case the object is already dropped before we t

Re: Avoid orphaned objects dependencies, take 3

2024-04-24 Thread Bertrand Drouvot
Hi, On Tue, Apr 23, 2024 at 04:20:46PM +, Bertrand Drouvot wrote: > Please find attached v2 that should not produce the issue anymore (I launched > a > lot of attempts without any issues). v1 was not strong enough as it was not > always checking for the dependent object exist

Re: Avoid orphaned objects dependencies, take 3

2024-04-23 Thread Bertrand Drouvot
Hi, On Tue, Apr 23, 2024 at 04:59:09AM +, Bertrand Drouvot wrote: > Hi, > > On Mon, Apr 22, 2024 at 03:00:00PM +0300, Alexander Lakhin wrote: > > 22.04.2024 13:52, Bertrand Drouvot wrote: > > > > > > That's weird, I just launched it several times with

Re: Avoid orphaned objects dependencies, take 3

2024-04-22 Thread Bertrand Drouvot
Hi, On Mon, Apr 22, 2024 at 03:00:00PM +0300, Alexander Lakhin wrote: > 22.04.2024 13:52, Bertrand Drouvot wrote: > > > > That's weird, I just launched it several times with the patch applied and > > I'm not > > able to see the seg fault (while I can see it constently

Re: Avoid orphaned objects dependencies, take 3

2024-04-22 Thread Bertrand Drouvot
Hi, On Mon, Apr 22, 2024 at 01:00:00PM +0300, Alexander Lakhin wrote: > Hi Bertrand, > > 22.04.2024 11:45, Bertrand Drouvot wrote: > > Hi, > > > > This new thread is a follow-up of [1] and [2]. > > > > Problem description: > > > > We ha

Re: Disallow changing slot's failover option in transaction block

2024-04-22 Thread Bertrand Drouvot
Hi, On Mon, Apr 22, 2024 at 11:32:14AM +0530, shveta malik wrote: > On Mon, Apr 22, 2024 at 5:57 AM Zhijie Hou (Fujitsu) > wrote: > > Attach the V3 patch which also addressed Shveta[1] and Bertrand[2]'s > > comments. Thanks! > Tested the patch, works well. Same here. R

Re: Patch to avoid orphaned dependencies

2024-04-22 Thread Bertrand Drouvot
cost (though not to zero). Thanks for the idea (and sorry for the delay replying to it)! I had a look at it and just created a new thread [1] based on your proposal. [1]: https://www.postgresql.org/message-id/flat/ZiYjn0eVc7pxVY45%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Ber

Avoid orphaned objects dependencies, take 3

2024-04-22 Thread Bertrand Drouvot
te that I'm not used to with this area of the code so that the patch might not take the option proposed by Tom the "right" way. Adding the patch to the July CF. Looking forward to your feedback, Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databa

Re: Disallow changing slot's failover option in transaction block

2024-04-19 Thread Bertrand Drouvot
so I think it needs a bit of rewording (as here it sounds like the sub is already created), , something like? "Always ensure the slot property failover matches the counterpart parameter of the subscription and vice versa." Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS

Re: promotion related handling in pg_sync_replication_slots()

2024-04-18 Thread Bertrand Drouvot
aid that, the logic that is in place to take care of the corner cases described up-thread seems reasonable to me. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-16 Thread Bertrand Drouvot
ase as well, SQL function will ultimately get error "cannot > synchronize replication slots concurrently", even though GUC is > enabled. Thus, I feel we should stick with samer error in all > scenarios. Okay, fine by me, let's forget about checking sync_replication_slots then. R

Re: Disallow changing slot's failover option in transaction block

2024-04-16 Thread Bertrand Drouvot
option as true cannot be executed inside a transaction block" Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-16 Thread Bertrand Drouvot
s() and emmit an error if sync_replication_slots is set to on? (The message could explicitly states that it's not possible to use the function if sync_replication_slots is set to on). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: promotion related handling in pg_sync_replication_slots()

2024-04-16 Thread Bertrand Drouvot
Hi, On Tue, Apr 16, 2024 at 08:21:04AM +0530, Amit Kapila wrote: > On Mon, Apr 15, 2024 at 7:47 PM Bertrand Drouvot > wrote: > > > > On Mon, Apr 15, 2024 at 06:29:49PM +0530, Amit Kapila wrote: > > > On Mon, Apr 15, 2024 at 6:21 PM Bertrand Drouvot > > > w

Re: promotion related handling in pg_sync_replication_slots()

2024-04-15 Thread Bertrand Drouvot
Hi, On Mon, Apr 15, 2024 at 06:29:49PM +0530, Amit Kapila wrote: > On Mon, Apr 15, 2024 at 6:21 PM Bertrand Drouvot > wrote: > > > > On Mon, Apr 15, 2024 at 03:38:53PM +0530, shveta malik wrote: > > > Now both worker and SQL > > > function set 'syncin

Re: promotion related handling in pg_sync_replication_slots()

2024-04-15 Thread Bertrand Drouvot
th worker and SQL > function set 'syncing' when they start and reset it when they exit. It means that it's not possible anymore to trigger a manual sync if sync_replication_slots is on. Indeed that would trigger: postgres=# select pg_sync_replication_slots(); ERROR: cannot synchronize repl

Re: Synchronizing slots from primary to standby

2024-04-06 Thread Bertrand Drouvot
Hi, On Sat, Apr 06, 2024 at 10:13:00AM +0530, Amit Kapila wrote: > On Fri, Apr 5, 2024 at 8:05 PM Bertrand Drouvot > wrote: > > I think the new LSN can be visible only when the corresponding WAL is > written by XLogWrite(). I don't know what in XLogSetAsyncXactLSN() can &g

Re: Synchronizing slots from primary to standby

2024-04-05 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 02:35:42PM +, Bertrand Drouvot wrote: > I think that maybe as a first step we should move the "elog(DEBUG2," message > as > proposed above to help debugging (that could help to confirm the above > theory). If you agree and think that m

Re: Synchronizing slots from primary to standby

2024-04-05 Thread Bertrand Drouvot
age as proposed above to help debugging (that could help to confirm the above theory). If the theory is proven then I'm not sure we need the extra complexity of injection point here, maybe just relying on the slots created via SQL API could be enough. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-04-05 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 04:09:01PM +0530, shveta malik wrote: > On Fri, Apr 5, 2024 at 10:09 AM Bertrand Drouvot > wrote: > > > > What about something like? > > > > ereport(LOG, > > errmsg("synchronized confirmed_flush_lsn for

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-05 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 11:21:43AM +0530, Bharath Rupireddy wrote: > On Wed, Apr 3, 2024 at 9:57 PM Bertrand Drouvot > wrote: > Please find the attached v36 patch. Thanks! A few comments: 1 === + +The timeout is measured from the time since the slot h

Re: Synchronizing slots from primary to standby

2024-04-04 Thread Bertrand Drouvot
Hi, On Fri, Apr 05, 2024 at 09:43:35AM +0530, shveta malik wrote: > On Fri, Apr 5, 2024 at 9:22 AM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Thu, Apr 04, 2024 at 05:31:45PM +0530, shveta malik wrote: > > > On Thu, Apr 4, 2024 at 2:

Re: Synchronizing slots from primary to standby

2024-04-04 Thread Bertrand Drouvot
) ", +remote_slot->name, + LSN_FORMAT_ARGS(slot->data.confirmed_flush), + LSN_FORMAT_ARGS(remote_slot->confirmed_lsn)); I don't think that the message is correct here. Unless I am mis

Re: Autogenerate some wait events code and documentation

2024-04-04 Thread Bertrand Drouvot
Hi, On Thu, Apr 04, 2024 at 07:14:47PM +0900, Michael Paquier wrote: > On Thu, Apr 04, 2024 at 09:28:36AM +0000, Bertrand Drouvot wrote: > > +# No "Backpatch" region here as code is generated automatically. > > > > What about "region here as has

Re: Autogenerate some wait events code and documentation

2024-04-04 Thread Bertrand Drouvot
Hi, On Thu, Apr 04, 2024 at 03:50:21PM +0900, Michael Paquier wrote: > On Tue, Mar 19, 2024 at 07:34:09AM +0000, Bertrand Drouvot wrote: > > I'm not sure as v2 used the "version >= 17" wording which I think would not > > need > > manual refresh each

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
Hi, On Wed, Apr 03, 2024 at 08:28:04PM +0530, Bharath Rupireddy wrote: > On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot > wrote: > > > > Just one comment on v32-0001: > > > > +# Synced slot on the standby must get its own inactive_since.

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
think we should just add some usleep() where appropriate and deny equality during the tests on inactive_since. Except for the above, v32-0001 LGTM. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
cation_slot_inactive_timeout/? CR9 === +# Start: Helper functions used for this test file +# End: Helper functions used for this test file I think that's the first TAP test with this comment. Not saying we should not but why did you feel the need to add those? [1]: https://www.postgresql.org/messa

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-03 Thread Bertrand Drouvot
=== + # Check that the captured time is sane + if (defined $reference_time) + { s/Check that the captured time is sane/Check that the inactive_since is sane/? Sorry if some of those comments could have been done while I did review v29-0001. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-04-02 Thread Bertrand Drouvot
Hi, On Tue, Apr 02, 2024 at 02:19:30PM +0530, Amit Kapila wrote: > On Tue, Apr 2, 2024 at 1:54 PM Bertrand Drouvot > wrote: > > What about adding a "wait" injection point in LogStandbySnapshot() to > > prevent > > checkpointer/bgwriter to log a standby snapshot

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-02 Thread Bertrand Drouvot
Hi, On Tue, Apr 02, 2024 at 12:41:35PM +0530, Bharath Rupireddy wrote: > On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot > wrote: > > > > > Or a simple solution is that the slotsync worker updates > > > inactive_since as it does for non-synced slots, and dis

Re: Synchronizing slots from primary to standby

2024-04-02 Thread Bertrand Drouvot
if (AmCheckpointerProcess() || AmBackgroundWriterProcess()) INJECTION_POINT("bgw-log-standby-snapshot"); And make use of it in the test, something like: $node_primary->safe_psql('postgres', "SELECT injection_points_attach('bgw-log-standby-s

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-02 Thread Bertrand Drouvot
r a simple solution is that the slotsync worker updates > inactive_since as it does for non-synced slots, and disables > timeout-based slot invalidation for synced slots. Yeah, I think the main question to help us decide is: do we want to invalidate "inactive" synced slots locally (in addition to synchronizing the invalidation from the primary)? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-04-01 Thread Bertrand Drouvot
Hi, On Tue, Apr 02, 2024 at 04:24:49AM +, Zhijie Hou (Fujitsu) wrote: > On Monday, April 1, 2024 9:28 PM Bertrand Drouvot > wrote: > > > > On Mon, Apr 01, 2024 at 05:04:53PM +0530, Amit Kapila wrote: > > > On Mon, Apr 1, 2024 at 2:51 PM Bertrand D

Re: Synchronizing slots from primary to standby

2024-04-01 Thread Bertrand Drouvot
Hi, On Mon, Apr 01, 2024 at 05:04:53PM +0530, Amit Kapila wrote: > On Mon, Apr 1, 2024 at 2:51 PM Bertrand Drouvot > wrote: > > Then there is no need to call WaitForStandbyConfirmation() as it could go > > until > > the RecoveryInProgress() in StandbySlotsHaveCaugh

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Bertrand Drouvot
Hi, On Sun, Mar 31, 2024 at 10:25:46AM +0530, Bharath Rupireddy wrote: > On Thu, Mar 28, 2024 at 3:13 PM Bertrand Drouvot > wrote: > > I think in this case it should always reflect the value from the primary (so > > that one can understand why it is invalidated). &

Re: Synchronizing slots from primary to standby

2024-04-01 Thread Bertrand Drouvot
alize() maintains a "small list" of "already serialized" snapshots. [1]: https://www.postgresql.org/message-id/ZgayTFIhLfzhpHci%40ip-10-97-1-34.eu-west-3.compute.internal Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Bertrand Drouvot
the data, and if the user needs helps to make sense of that data, then give them that help separately. " [1]: https://www.postgresql.org/message-id/CAA4eK1JtKieWMivbswYg5FVVB5FugCftLvQKVsxh%3Dm_8nk04vw%40mail.gmail.com [2]: https://www.postgresql.org/message-id/CA%2BTgmob_Ta-t2ty8QrKHBGnNLrf4ZYcwhGHGFsuUoFrAEDw4sA%40mail.gmail.com Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-04-01 Thread Bertrand Drouvot
Hi, On Mon, Apr 01, 2024 at 09:04:43AM +0530, Amit Kapila wrote: > On Fri, Mar 29, 2024 at 6:17 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 29, 2024 at 03:03:01PM +0530, Amit Kapila wrote: > > > On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot > > > w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 03:03:01PM +0530, Amit Kapila wrote: > On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot > wrote: > > > > On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote: > > > > > > Commit message states: "why we can't just upd

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 02:35:22PM +0530, Amit Kapila wrote: > On Fri, Mar 29, 2024 at 1:08 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 29, 2024 at 07:23:11AM +, Zhijie Hou (Fujitsu) wrote: > > > On Friday, March 29, 2024 2:48 PM Bertrand Drouvot >

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
Hi, On Fri, Mar 29, 2024 at 07:23:11AM +, Zhijie Hou (Fujitsu) wrote: > On Friday, March 29, 2024 2:48 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Fri, Mar 29, 2024 at 01:06:15AM +, Zhijie Hou (Fujitsu) wrote: > > > Attach a new vers

Re: Synchronizing slots from primary to standby

2024-03-29 Thread Bertrand Drouvot
n this hidden slot only to advance to the max lsn of the synced slots I'm not sure that would be enough, just asking your thoughts on this (benefits would be to avoid calling pg_logical_replication_slot_advance() on each sync slots and during the sync cycles). Regards, -- Bertrand Drouvot Pos

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-29 Thread Bertrand Drouvot
t. Right, but for a very active slot it is likely that we call GetCurrentTimestamp() during almost each sync cycle. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Synchronizing slots from primary to standby

2024-03-28 Thread Bertrand Drouvot
Hi, On Thu, Mar 28, 2024 at 05:05:35PM +0530, Amit Kapila wrote: > On Thu, Mar 28, 2024 at 3:34 PM Bertrand Drouvot > wrote: > > > > On Thu, Mar 28, 2024 at 04:38:19AM +, Zhijie Hou (Fujitsu) wrote: > > > > > To fix this, we could use the fast fo

Re: Synchronizing slots from primary to standby

2024-03-28 Thread Bertrand Drouvot
ndering about the performance impact (even in fast_forward mode), might be worth to keep an eye on it. Should we create a 17 open item [1]? [1]: https://wiki.postgresql.org/wiki/PostgreSQL_17_Open_Items Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-28 Thread Bertrand Drouvot
ePossiblyObsoleteSlot() is not called with such a reason, better to use an Assert here and in the caller too? CR11 === +++ b/src/test/recovery/t/050_invalidate_slots.pl why not using 019_replslot_limit.pl? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-27 Thread Bertrand Drouvot
Hi, On Wed, Mar 27, 2024 at 09:00:37PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 27, 2024 at 6:54 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote: > > > On Wed, Mar 27, 2024 at 3:

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-27 Thread Bertrand Drouvot
Hi, On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot > Please see the attached v28 patch. Thanks! 1 === sorry I missed it in the previous review if (!(RecoveryInProgress() && slo

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-27 Thread Bertrand Drouvot
ts) will remain zero until the next synchronization. 2 === +=item $node->create_logical_slot_on_standby(self, primary, slot_name, dbname) get_slot_inactive_since_value instead? 3 === +against given reference time. s/given reference/optional given reference/? Apart from the above, LGTM. R

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-27 Thread Bertrand Drouvot
x_slot_xid_age = 1.5 billion or so and also set > replication_slot_inactive_timeout = 2 days or so to make the database > foolproof. Yeah, I think that both makes senses. The reason is that one depends of the database activity and slot activity (the xid age one) while the other (the timeout one) depends only of the slot activity. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
Hi, On Wed, Mar 27, 2024 at 10:08:33AM +0530, Bharath Rupireddy wrote: > On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot > wrote: > > > > - if (!(RecoveryInProgress() && slot->data.synced)) > > + if

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
nce $slot_creation_time\n"; garbage? 7 === +# Capture and validate inactive_since of a given slot. +sub capture_and_validate_slot_inactive_since +{ + my ($node, $slot_name, $slot_creation_time) = @_; + my $name = $node->name; We know have capture_and_validate_slot_inactive_since at 2 places: 040_standby_failo

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-26 Thread Bertrand Drouvot
r not) a slot based one (in addition to the GUC). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
Hi, On Tue, Mar 26, 2024 at 04:17:53PM +0530, shveta malik wrote: > On Tue, Mar 26, 2024 at 3:50 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > > I think there may have been some misunderstanding here. > > > > Indeed ;-) > > > > > But

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
e fine if we just sync the value from > > primary and document that fact? After the promotion, we can reset it > > to the current time so that it gets its own time. Do you see any > > issues with it? > > Yes, we can do that. But curious to know, do we see any additional

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
Hi, On Tue, Mar 26, 2024 at 03:17:36PM +0530, shveta malik wrote: > On Tue, Mar 26, 2024 at 1:54 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote: > > > On Tue, Mar 26, 2024 at 1:15 PM Bertrand

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
ut moving "now = GetCurrentTimestamp()" outside of the for loop? (it would be less costly and probably good enough). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-26 Thread Bertrand Drouvot
elease the sync slots. So at the time of release, > inactive_since will be updated. See email [1]. I don't think we should set inactive_since to the current time at each sync cycle, see [1] as to why. What do you think? [1]: https://www.postgresql.org/message-id/ZgKGIDC5lttWTdJH%40ip-10-97-1-

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
Hi, On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote: > On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot > wrote: > > > > 2 === > > > > It looks like inactive_since is set to the current timestamp on the standby > > each time the sync

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
ostgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; slot_name |inactive_since -+--- lsub27_slot | 2024-03-26 07:43:56.387324+00 lsub28_slot | 2024-03-26 07:43:56.387338+00 I don't think that should be the case. Reg

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-26 Thread Bertrand Drouvot
Hi, On Tue, Mar 26, 2024 at 05:55:11AM +, Bertrand Drouvot wrote: > Hi, > > On Tue, Mar 26, 2024 at 09:30:32AM +0530, shveta malik wrote: > > On Mon, Mar 25, 2024 at 12:43 PM shveta malik > > wrote: > > > > > > I have one concern, for sync

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-25 Thread Bertrand Drouvot
s that if the primary is down then one could look at the synced "active_since" on the standby to get an idea of it (depends of the last sync though). The issue that I can see with your proposal is: what if one synced the slots manually (with pg_sync_replication_slots()) but does not use the

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-25 Thread Bertrand Drouvot
Hi, On Mon, Mar 25, 2024 at 12:25:37PM -0400, Robert Haas wrote: > On Mon, Mar 25, 2024 at 12:12 PM Bertrand Drouvot > wrote: > > Would "released_time" sounds better? (at the end this is exactly what it > > does > > represent unless for the case where

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-25 Thread Bertrand Drouvot
mentioned by Shveta in [1] during promotion. [1]: https://www.postgresql.org/message-id/CAJpy0uCLu%2BmqAwAMum%3DpXE9YYsy0BE7hOSw_Wno5vjwpFY%3D63g%40mail.gmail.com Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-25 Thread Bertrand Drouvot
Hi, On Mon, Mar 25, 2024 at 11:49:00AM -0400, Robert Haas wrote: > On Mon, Mar 25, 2024 at 11:16 AM Bertrand Drouvot > wrote: > > > IIUC, Bertrand's point was that users can interpret last_active_time > > > as a value that gets updated each time they decode a change wh

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-25 Thread Bertrand Drouvot
Hi, On Mon, Mar 25, 2024 at 08:59:55PM +0530, Amit Kapila wrote: > On Mon, Mar 25, 2024 at 8:46 PM Bertrand Drouvot > wrote: > > > > On Mon, Mar 25, 2024 at 08:38:16PM +0530, Amit Kapila wrote: > > > On Mon, Mar 25, 2024 at 7:51 PM Robert Haas wrote: > > >

Re: pgsql: Track last_inactive_time in pg_replication_slots.

2024-03-25 Thread Bertrand Drouvot
information as well. Now, if the server > is down for a long time, we may invalidate the slots as soon as the > server comes up. So, instead, we just set this field at the time we > read slots for disk and then reset it to 0/NULL as soon as the slot > became active. Right, and we

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-25 Thread Bertrand Drouvot
Hi, On Mon, Mar 25, 2024 at 02:39:50PM +0530, shveta malik wrote: > I am listing the concerns raised by me: > 3) alter replication slot to alter inactive_timout for synced slots on > standby, should this be allowed? I don't think it should be allowed. Regards, -- Bertrand Drouvot P

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-25 Thread Bertrand Drouvot
Hi, On Mon, Mar 25, 2024 at 02:07:21PM +0530, shveta malik wrote: > On Mon, Mar 25, 2024 at 1:37 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Mon, Mar 25, 2024 at 12:59:52PM +0530, Amit Kapila wrote: > > > On Mon, Mar 25, 2024 at 12:43 PM shveta mali

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-25 Thread Bertrand Drouvot
such slots are not usable anyway (until the standby gets promoted). So, I think that last_inactive_time does not make sense if the slot never had the chance to be active. OTOH I think the timeout invalidation (if any) should be synced from primary. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-25 Thread Bertrand Drouvot
e made minor changes in the comments. See and > > let me know what you think of attached. > Thanks! v19-0001 LGTM, just one Nit comment for 019_replslot_limit.pl: The code for "Get last_inactive_time value after the slot's creation" and "Check that the captured time is

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-23 Thread Bertrand Drouvot
one testing the timeout. Could be added in one of the existing .pl like 001_stream_rep.pl for example. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-22 Thread Bertrand Drouvot
Hi, On Fri, Mar 22, 2024 at 06:02:11PM +0530, Amit Kapila wrote: > On Fri, Mar 22, 2024 at 5:30 PM Bertrand Drouvot > wrote: > > On Fri, Mar 22, 2024 at 03:56:23PM +0530, Amit Kapila wrote: > > > > > > > > That would avoid testing twice "slot-&g

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-22 Thread Bertrand Drouvot
Hi, On Fri, Mar 22, 2024 at 04:16:19PM +0530, Amit Kapila wrote: > On Fri, Mar 22, 2024 at 3:23 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 22, 2024 at 02:59:21PM +0530, Amit Kapila wrote: > > > On Fri, Mar 22, 2024 at 2:27 PM Bertrand Drouvot > > > w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-22 Thread Bertrand Drouvot
Hi, On Fri, Mar 22, 2024 at 03:56:23PM +0530, Amit Kapila wrote: > On Fri, Mar 22, 2024 at 3:15 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote: > > > > 1 === > > > > @@ -69

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-22 Thread Bertrand Drouvot
Hi, On Fri, Mar 22, 2024 at 02:59:21PM +0530, Amit Kapila wrote: > On Fri, Mar 22, 2024 at 2:27 PM Bertrand Drouvot > wrote: > > > > On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote: > > > > > 0001 Track invalidation_reason in pg_r

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-22 Thread Bertrand Drouvot
Hi, On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote: > On Fri, Mar 22, 2024 at 12:39 PM Bertrand Drouvot > wrote: > > > > > > Please find the v14-0001 patch for now. > > > > Thanks! > > > > > LGTM. Let's wait

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-22 Thread Bertrand Drouvot
Hi, On Fri, Mar 22, 2024 at 01:45:01PM +0530, Bharath Rupireddy wrote: > On Fri, Mar 22, 2024 at 12:39 PM Bertrand Drouvot > wrote: > > > > > > Please find the v14-0001 patch for now. > > > > Thanks! > > > > > LGTM. Let's wait

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-22 Thread Bertrand Drouvot
d then I'll push it. LGTM too. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-21 Thread Bertrand Drouvot
} " while we set set_last_inactive_at to false at creation time so that last_inactive_at is not set to GetCurrentTimestamp(). We should set set_last_inactive_at to true if a timeout is provided during the slot creation. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-21 Thread Bertrand Drouvot
Hi, On Thu, Mar 21, 2024 at 05:05:46AM +0530, Bharath Rupireddy wrote: > On Wed, Mar 20, 2024 at 1:04 PM Bertrand Drouvot > wrote: > > > > On Wed, Mar 20, 2024 at 08:58:05AM +0530, Amit Kapila wrote: > > > On Wed, Mar 20, 2024 at 12:49 AM Bharath Rupireddy > > &g

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-21 Thread Bertrand Drouvot
Hi, On Thu, Mar 21, 2024 at 11:53:32AM +0530, Amit Kapila wrote: > On Thu, Mar 21, 2024 at 11:37 AM Bertrand Drouvot > wrote: > > > > We could also do the above 3 and altering the timeout with a replication > > connection but the SQL API seems more natural to me. &

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-21 Thread Bertrand Drouvot
Hi, On Thu, Mar 21, 2024 at 11:43:54AM +0530, Amit Kapila wrote: > On Thu, Mar 21, 2024 at 11:23 AM Bertrand Drouvot > wrote: > > > > On Thu, Mar 21, 2024 at 08:47:18AM +0530, Amit Kapila wrote: > > > On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot > > > w

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-21 Thread Bertrand Drouvot
and via > create/alter subscription respectively, and implement > pg_alter_replication_slot(). +1 on this. > FWIW, adding the new SQL API pg_alter_replication_slot() isn't that hard. Also I think we should ensure that one could "only" alter the timeout property for the time being (if not that could lead to the subscription inconsistency mentioned above). Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-20 Thread Bertrand Drouvot
Hi, On Thu, Mar 21, 2024 at 08:47:18AM +0530, Amit Kapila wrote: > On Wed, Mar 20, 2024 at 1:51 PM Bertrand Drouvot > wrote: > > > > On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote: > > > > > > 2. last_inactive_at and inactiv

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-20 Thread Bertrand Drouvot
Hi, On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote: > On Mon, Mar 18, 2024 at 3:02 PM Bertrand Drouvot > wrote: > > > > > > Hm. Are you suggesting inactive_timeout to be a slot level parameter > > > > similar t

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-20 Thread Bertrand Drouvot
Hi, On Wed, Mar 20, 2024 at 12:48:55AM +0530, Bharath Rupireddy wrote: > On Mon, Mar 18, 2024 at 3:02 PM Bertrand Drouvot > wrote: > > > > > > Hm. Are you suggesting inactive_timeout to be a slot level parameter > > > > similar t

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-20 Thread Bertrand Drouvot
, so that queries would return the right invalidation status. > > 4. New invalidation mechanisms interaction with slot sync feature. > > > > Yeah, this is important. My initial thoughts are that synced slots > shouldn't be invalidated on the standby due to timeout. +1 Rega

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-19 Thread Bertrand Drouvot
Hi, On Tue, Mar 19, 2024 at 04:20:35PM +0530, Amit Kapila wrote: > On Tue, Mar 19, 2024 at 3:11 PM Bertrand Drouvot > wrote: > > > > On Tue, Mar 19, 2024 at 10:56:25AM +0530, Amit Kapila wrote: > > > On Mon, Mar 18, 2024 at 8:19 PM Bertrand Drouvot > > >

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-19 Thread Bertrand Drouvot
Hi, On Tue, Mar 19, 2024 at 10:56:25AM +0530, Amit Kapila wrote: > On Mon, Mar 18, 2024 at 8:19 PM Bertrand Drouvot > wrote: > > Agree. While it makes sense to invalidate slots for wal removal in > > CreateCheckPoint() (because this is the place where wal is removed), I 'm

Re: Autogenerate some wait events code and documentation

2024-03-19 Thread Bertrand Drouvot
Hi, On Tue, Mar 19, 2024 at 09:59:35AM +0900, Michael Paquier wrote: > On Mon, Mar 18, 2024 at 05:57:02PM +0000, Bertrand Drouvot wrote: > > Thanks for looking at it! > > Oh right, the comment is wrong, re-worded in v2 attached. > > I've added a couple of fake

Re: Autogenerate some wait events code and documentation

2024-03-18 Thread Bertrand Drouvot
Hi, On Mon, Mar 18, 2024 at 08:49:34AM -0700, Noah Misch wrote: > On Mon, Mar 18, 2024 at 09:04:44AM +0000, Bertrand Drouvot wrote: > > --- a/src/backend/utils/activity/wait_event_names.txt > > +++ b/src/backend/utils/activity/wait_event_names.txt > > @@ -24,7 +24,12 @@ &g

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-18 Thread Bertrand Drouvot
em when?: - their usage resume - in pg_get_replication_slots() The idea is to invalidate the slot when one resumes activity on it or wants to get information about it (and among other things wants to know if the slot is valid or not). Thoughts? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-18 Thread Bertrand Drouvot
replication slot to s/a invalidation/an invalidation/? 4 === While at it, shouldn't we also rename "conflict" to say "invalidation_cause" in InvalidatePossiblyObsoleteSlot()? 5 === + * rows_removed and wal_level_insufficient are only two reasons s/are only two/are the only t

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-03-18 Thread Bertrand Drouvot
to some plugins). As far max_slot_xid_age I've the feeling that a new GUC is good enough. > > Alternatively, we can go the route of making GUC a list of key-value > > pairs of {slot_name, inactive_timeout}, but this kind of GUC for > > setting slot level parameters is going to be the firs

  1   2   3   >