2025-08-14 release announcement draft

2025-08-11 Thread Noah Misch
Attached is a draft of the 2025-08-14 update release + beta3 announcement. Please provide feedback no later than 2025-08-14 00:00 UTC. (Jonathan Katz would normally be collecting announcement feedback, but I'm covering for a scheduling conflict.) Thanks, nm The PostgreSQL Global Development Grou

Re: BackgroundPsql swallowing errors on windows

2025-08-10 Thread Noah Misch
On Sun, Mar 09, 2025 at 10:23:55AM -0700, Noah Misch wrote: > On Sun, Mar 09, 2025 at 12:47:34PM -0400, Andres Freund wrote: > > On 2025-02-16 17:52:36 -0800, Noah Misch wrote: > > > On Sun, Feb 16, 2025 at 08:42:50PM -0500, Andres Freund wrote: > > > > On Februar

Re: Test instability when pg_dump orders by OID

2025-08-10 Thread Noah Misch
On Sun, Aug 10, 2025 at 04:41:20PM +0500, Kirill Reshke wrote: > On Sun, 10 Aug 2025 at 12:00, Alexander Lakhin wrote: > > 04.08.2025 03:03, Noah Misch wrote: > > > Pushed as 0decd5e. ... > > > > Please look at a new anomaly introduced with that commit. The foll

Re: A few patches to clarify snapshot management

2025-08-09 Thread Noah Misch
On Tue, Jan 07, 2025 at 11:55:00AM +0200, Heikki Linnakangas wrote: > On 07/01/2025 00:00, Andres Freund wrote: > > On 2024-12-20 19:31:01 +0200, Heikki Linnakangas wrote: > > > While playing around some more with this, I noticed that this code in > > > GetTransactionSnapshot() is never reached, an

Re: Test instability when pg_dump orders by OID

2025-08-03 Thread Noah Misch
On Fri, Jul 25, 2025 at 02:01:01PM -0400, Robert Haas wrote: > On Thu, Jul 24, 2025 at 10:27 PM Noah Misch wrote: > > I regret missing those in v1. I've attached v2, including branch-specific > > patches. I'll first need to back-patch 350e6b8, which fixed sorting of &g

Re: Non-text mode for pg_dumpall

2025-07-29 Thread Noah Misch
On Tue, Jul 29, 2025 at 04:09:13PM -0400, Andrew Dunstan wrote: > here's a reversion patch for master. > This reverts parts or all of the following commits: I briefly looked through this. The biggest non-reverted part is, I think, c1da728 "Move common pg_dump code related to connections to a

Re: Non-text mode for pg_dumpall

2025-07-27 Thread Noah Misch
On Fri, Jul 25, 2025 at 04:59:29PM -0400, Tom Lane wrote: > Andrew Dunstan writes: > > Before we throw the baby out with the bathwater, how about this > > suggestion? pg_dumpall would continue to produce globals.dat, but it > > wouldn't be processed by pg_restore, which would only restore the >

Re: Non-text mode for pg_dumpall

2025-07-25 Thread Noah Misch
On Thu, Jul 24, 2025 at 04:33:15PM -0400, Andrew Dunstan wrote: > On 2025-07-21 Mo 8:53 PM, Noah Misch wrote: > > I suspect this is going to end with a structured dump like we use on the > > pg_dump (per-database) side. It's not an accident that v17 pg_restore > > doe

Re: Test instability when pg_dump orders by OID

2025-07-24 Thread Noah Misch
On Mon, Jul 21, 2025 at 09:40:02AM -0400, Robert Haas wrote: > On Fri, Jul 18, 2025 at 3:17 PM Noah Misch wrote: > > +* Sort by encoding, per pg_collation_name_enc_nsp_index. > > Wherever > > +* this changes dump order, restoring the d

Re: Non-text mode for pg_dumpall

2025-07-24 Thread Noah Misch
On Thu, Jul 17, 2025 at 03:46:41PM +0530, Mahendra Singh Thalor wrote: > On Wed, 16 Jul 2025 at 05:50, Noah Misch wrote: > > On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote: > > > On Wed, 9 Jul 2025 at 02:58, Noah Misch wrote: > > > > On F

Re: Non-text mode for pg_dumpall

2025-07-21 Thread Noah Misch
On Mon, Jul 21, 2025 at 04:41:03PM -0400, Andrew Dunstan wrote: > On 2025-07-17 Th 6:18 AM, Mahendra Singh Thalor wrote > > > > > > > --- a/src/bin/pg_dump/pg_restore.c > > > > > > > +++ b/src/bin/pg_dump/pg_restore.c > > > > > > > +/* > > > > > > > + * read_one_statement > > > > > > > + * > > > >

Re: Test instability when pg_dump orders by OID

2025-07-18 Thread Noah Misch
On Thu, Jul 17, 2025 at 09:24:02AM -0400, Robert Haas wrote: > On Mon, Jul 7, 2025 at 3:27 PM Noah Misch wrote: > > Let's get rid of pg_dump's need to sort by OID, apart from catalog > > corruption > > scenarios. > > +1. I had at one point believed that so

Re: Non-text mode for pg_dumpall

2025-07-15 Thread Noah Misch
On Thu, Jul 10, 2025 at 12:21:03AM +0530, Mahendra Singh Thalor wrote: > On Wed, 9 Jul 2025 at 02:58, Noah Misch wrote: > > On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote: > > > Thanks. I have pushed these now with a few further small tweaks. > > >

Re: Can can I make an injection point wait occur no more than once?

2025-07-09 Thread Noah Misch
On Tue, Jul 08, 2025 at 11:43:17PM -0400, Peter Geoghegan wrote: > On Tue, Jul 8, 2025 at 11:04 PM Noah Misch wrote: > > > -backwards_scan_session: NOTICE: notice triggered for injection point > > > lock-and-validate-new-lastcurrblkno > > > +ERROR: could not

Re: Can can I make an injection point wait occur no more than once?

2025-07-08 Thread Noah Misch
On Tue, Jul 08, 2025 at 11:21:20AM -0400, Peter Geoghegan wrote: > On Mon, Jul 7, 2025 at 9:53 PM Noah Misch wrote: > > If it continues to be a problem, consider sharing the patch that's behaving > > this way for you. > > Attached patch shows my current progress with th

Re: [PATCH] Add support for displaying database service in psql prompt

2025-07-08 Thread Noah Misch
On Tue, Jul 08, 2025 at 08:52:08AM +0900, Michael Paquier wrote: > On Sun, Jul 06, 2025 at 08:00:09PM -0700, Noah Misch wrote: > > I think the choice to make there is whether to call PQconninfo() once per > > prompt emission or to cache the value, invalidating that cach

Re: Non-text mode for pg_dumpall

2025-07-08 Thread Noah Misch
On Fri, Apr 04, 2025 at 04:11:05PM -0400, Andrew Dunstan wrote: > Thanks. I have pushed these now with a few further small tweaks. This drops all databases: pg_dumpall --clean -Fd -f /tmp/dump pg_restore -d template1 --globals-only /tmp/dump That didn't match my expectations given this help text

Re: Can can I make an injection point wait occur no more than once?

2025-07-07 Thread Noah Misch
On Mon, Jul 07, 2025 at 09:40:20PM -0400, Peter Geoghegan wrote: > I'm finding that the FreeBSD Meson CI target consistently fails with > this setup, though. And with just about any variant I can think of; > seems to fail quite reliably. The initial SELECT backwards scan > statement will complete w

Re: Can can I make an injection point wait occur no more than once?

2025-07-07 Thread Noah Misch
On Mon, Jul 07, 2025 at 05:31:30PM -0400, Peter Geoghegan wrote: > I have successfully simulated "wait no more than once" by adding C > code to nbtree that looks like this: > > if (likely(!P_ISDELETED(opaque) && >opaque->btpo_next == lastcurrblkno)) >

Test instability when pg_dump orders by OID

2025-07-07 Thread Noah Misch
ty was always bad for users, but the test brought it to the forefront. One might argue for back-patching $SUBJECT further, too. Thanks, nm From: Noah Misch Demo 002_pg_upgrade.pl schema diffs from using OID in pg_dump sort. Not for commit. This reverses the pg_dump OID sort for one side

Re: [PATCH] Add support for displaying database service in psql prompt

2025-07-06 Thread Noah Misch
On Mon, Jul 07, 2025 at 11:06:06AM +0900, Michael Paquier wrote: > On Sun, Jul 06, 2025 at 09:13:19AM -0700, Noah Misch wrote: > > This new PQservice() function from commit 4b99fed75 came up in the annual > > exports.txt diff. The standard in libpq has been to not clutter th

Re: Issues with 2PC at recovery: CLOG lookups and GlobalTransactionData

2025-07-06 Thread Noah Misch
On Fri, Jun 20, 2025 at 09:02:00AM +0900, Michael Paquier wrote: > On Fri, Jun 06, 2025 at 03:34:13PM -0700, Noah Misch wrote: > > Shared memory should (not "must") omit a GXACT whose prepare_start_lsn lies > > beyond the point recovery has reached. In other wo

Re: [PATCH] Add support for displaying database service in psql prompt

2025-07-06 Thread Noah Misch
On Wed, Dec 18, 2024 at 03:17:36PM +0900, Michael Paquier wrote: > Done. This new PQservice() function from commit 4b99fed75 came up in the annual exports.txt diff. The standard in libpq has been to not clutter the API with new functions that simply retrieve one PQconninfoOption value. PQconninf

Re: Get rid of WALBufMappingLock

2025-07-04 Thread Noah Misch
ing, so I made the attached local edits in the course of the review. I don't know whether my choices are better or just tailored to my biases. If someone thinks this would improve the tree, I can polish this and commit it. Absent that, consider these notes to myself and to other post-commit revi

Re: Non-blocking archiver process

2025-07-04 Thread Noah Misch
On Fri, Jul 04, 2025 at 08:46:08AM +0200, Ronan Dunklau wrote: > We've noticed a behavior that seems surprising to us. > Since DROP DATABASE now waits for a ProcSignalBarrier, it can hang up > indefinitely if the archive_command hangs. > > The reason for this is that the builtin archive module

walwriter can set XLP_BKP_REMOVABLE wrongly: race w/ backup start

2025-07-04 Thread Noah Misch
h?q=XLP_BKP_REMOVABLE&literal=1&perpkg=1 finds none, and 2016-2018 discussion postgr.es/m/flat/579297F8.7020107%40anastigmatix.net found none. I wonder if ceasing to set XLP_BKP_REMOVABLE for v19 is the best fix. I don't plan to address this myself, so I'm leaving this here.

Patch review marketplace

2025-06-28 Thread Noah Misch
At https://wiki.postgresql.org/wiki/PGConf.dev_2025_Developer_Meeting I proposed a "review marketplace" experiment. Feedback at that developer meeting improved on the proposal. I've implemented it at https://wiki.postgresql.org/wiki/Review_Marketplace and have seeded the offer book with one offer

Re: Proper object locking for GRANT/REVOKE

2025-06-23 Thread Noah Misch
On Wed, Jun 11, 2025 at 05:22:53PM +0200, Peter Eisentraut wrote: > On 09.12.24 02:25, Noah Misch wrote: > > > Ok, we should probably put that comment back in slightly altered form, > > > like > > > > > > "XXX This function intentionally takes only an

Re: Issues with 2PC at recovery: CLOG lookups and GlobalTransactionData

2025-06-06 Thread Noah Misch
On Thu, Jun 05, 2025 at 04:22:48PM +0900, Michael Paquier wrote: > On Mon, Jun 02, 2025 at 06:48:46PM -0700, Noah Misch wrote: > > The wasShutdown case reaches consistency from the beginning, so I don't see > > that as an example of a time we benefit from reading pg_twophase

Re: Wrong security context for deferred triggers?

2025-06-05 Thread Noah Misch
On Thu, Jun 05, 2025 at 11:10:04AM -0400, Tom Lane wrote: > Noah Misch writes: > > Phrase "the role that executed the statement" doesn't match what happens if > > the role changes mid-statement. Example of a statement that does so: > > > select set_conf

Re: PG 18 release notes draft committed

2025-06-04 Thread Noah Misch
On Wed, Jun 04, 2025 at 06:10:38PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > I think we have to keep the non-SECURITY-DEFINER designation to keep the > > text accurate, I don't see it that way, because there's currently no experiment the user can perform to distinguish between the follow

Re: PG 18 release notes draft committed

2025-06-04 Thread Noah Misch
On Wed, Jun 04, 2025 at 04:45:18PM -0400, Bruce Momjian wrote: > On Tue, Jun 3, 2025 at 10:21:23AM -0700, Noah Misch wrote: > > When a commit changes the user that runs a function in existing queries, I > > think that almost always needs a release notes entry. It would follow

Re: PG 18 release notes draft committed

2025-06-03 Thread Noah Misch
On Thu, May 01, 2025 at 10:44:50PM -0400, Bruce Momjian wrote: > I have committd the first draft of the PG 18 release notes. When a commit changes the user that runs a function in existing queries, I think that almost always needs a release notes entry. It would follow that commit 01463e1 needs a

Re: Issues with 2PC at recovery: CLOG lookups and GlobalTransactionData

2025-06-02 Thread Noah Misch
On Fri, May 09, 2025 at 02:08:26PM +0900, Michael Paquier wrote: > On Tue, Feb 18, 2025 at 04:57:47PM -0800, Noah Misch wrote: > > As I wrote in [1], "By the time we reach consistency, every file in > > pg_twophase will be applicable (not committed or aborted)." If we fin

Re: Inval reliability, especially for inplace updates

2025-05-10 Thread Noah Misch
On Sun, Nov 03, 2024 at 10:29:25AM -0800, Noah Misch wrote: > On Fri, Nov 01, 2024 at 04:38:29PM -0700, Noah Misch wrote: > > This was a near miss to having a worst-in-years regression in a minor > > release, > > so I'm proposing this sequence: > > > > - R

Re: AIO v2.5

2025-05-08 Thread Noah Misch
On Thu, May 08, 2025 at 09:06:18PM -0400, Andres Freund wrote: > On 2025-05-02 20:05:11 -0700, Noah Misch wrote: > > On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote: > We do need to hold interrupts in a few other places, I think - with some debug > infrastructu

Re: AIO v2.5

2025-05-02 Thread Noah Misch
On Wed, Apr 30, 2025 at 04:00:35PM -0400, Andres Freund wrote: > pgaio_io_wait_for_free() does what it says on the tin. For that, after a bunch > of other things, finds the oldest in-flight IO and waits for it. > > PgAioHandle *ioh = dclist_head_element(PgAioHandle, node, >

Re: dispchar for oauth_client_secret

2025-04-28 Thread Noah Misch
On Mon, Apr 21, 2025 at 08:18:58AM -0700, Jacob Champion wrote: > Attached is my proposed fix. 0001 disables use of the new oauth_* > options in our FDWs. 0002 changes dispchar. > Subject: [PATCH 1/2] oauth: Disallow OAuth connections via postgres_fdw/dblink > Subject: [PATCH 2/2] oauth: Classify

Re: type cache cleanup improvements

2025-04-28 Thread Noah Misch
On Mon, Apr 21, 2025 at 04:54:08AM +0300, Alexander Korotkov wrote: > On Sat, Apr 12, 2025 at 12:43 AM Alexander Korotkov > wrote: > > On Fri, Apr 11, 2025 at 11:32 PM Noah Misch wrote: > > > On Tue, Oct 22, 2024 at 08:33:24PM +0300, Alexander Korotkov wrote: > > &g

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2025-04-26 Thread Noah Misch
On Fri, Apr 25, 2025 at 03:35:06PM -0400, Andres Freund wrote: > On 2025-04-20 14:53:39 -0700, Noah Misch wrote: > > The checkpoints and WAL creation took 30s, but archiving was only 20% done > > (based on file name 0001006D) at the 360s PGCTLTIMEOUT. >

Re: pg_upgrade-breaking release

2025-04-24 Thread Noah Misch
On Thu, Apr 24, 2025 at 08:37:56AM -0400, Bruce Momjian wrote: > On Thu, Apr 24, 2025 at 08:35:10AM -0400, Greg Sabino Mullane wrote: > > On Thu, Apr 24, 2025 at 8:12 AM Bruce Momjian wrote: > > > > Do we think most people are _not_ going to use pg_upgrade now that we > > are defaulting t

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2025-04-20 Thread Noah Misch
On Sun, Apr 20, 2025 at 02:53:39PM -0700, Noah Misch wrote: > On Mon, Apr 14, 2025 at 09:19:35AM +0900, Michael Paquier wrote: > > On Sun, Apr 13, 2025 at 11:51:57AM -0400, Tom Lane wrote: > > > Noah Misch writes: > > > > Tom and Michael, do you still object to

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2025-04-20 Thread Noah Misch
On Mon, Apr 14, 2025 at 09:19:35AM +0900, Michael Paquier wrote: > On Sun, Apr 13, 2025 at 11:51:57AM -0400, Tom Lane wrote: > > Noah Misch writes: > > > Tom and Michael, do you still object to the test addition, or not? If > > > there > > > are no new or re

Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

2025-04-20 Thread Noah Misch
On Sat, Apr 19, 2025 at 12:30:57PM -0700, Jeff Davis wrote: > On Thu, 2025-04-17 at 06:58 -0700, Noah Misch wrote: > > Should initcap_wbnext() pass in a locale-dependent "bool posix" > > argument like > > the others calls the commit changed? > > Yes, I bel

Re: Unicode full case mapping: PG_UNICODE_FAST, and standard-compliant UCS_BASIC

2025-04-17 Thread Noah Misch
On Fri, Jan 17, 2025 at 04:06:20PM -0800, Jeff Davis wrote: > Committed 0001 and 0002. > Upon reviewing the discussion threads, I removed the Unicode "adjust to > Cased" behavior when titlecasing. As Peter pointed out[1], it doesn't > match the documentation or expectations for INITCAP(). While c

Re: pipelining in psql, commit 41625ab

2025-04-16 Thread Noah Misch
On Wed, Apr 16, 2025 at 09:46:42AM -0700, Michael Paquier wrote: > On Tue, Apr 15, 2025 at 02:34:50PM -0700, Noah Misch wrote: > > On Fri, Feb 21, 2025 at 11:33:41AM +0900, Michael Paquier wrote: > > commit 41625ab wrote: > >> --- a/src/bin/psql/help.c > &

Re: [18] Unintentional behavior change in commit e9931bfb75

2025-04-15 Thread Noah Misch
On Tue, Apr 15, 2025 at 04:08:55PM -0700, Jeff Davis wrote: > On Mon, 2025-04-14 at 13:44 -0700, Noah Misch wrote: > > However, I think v17's > > concept of separate PG_REGEX_ symbols for the default-locale case is > > still the > > right thing for v18.  In other w

pipelining in psql, commit 41625ab

2025-04-15 Thread Noah Misch
On Fri, Feb 21, 2025 at 11:33:41AM +0900, Michael Paquier wrote: > So let's take one step here, I have applied the main patch. commit 41625ab wrote: > * \syncpipeline queues a synchronisation request, without flushing the > commands to the server, equivalent of PQsendPipelineSync(). libpq

Re: dispchar for oauth_client_secret

2025-04-15 Thread Noah Misch
On Tue, Apr 15, 2025 at 01:16:12PM -0700, Jacob Champion wrote: > On Tue, Apr 15, 2025 at 12:14 PM Noah Misch wrote: > > I suspect this should use .dispchar="*" to encourage UIs to display > > oauth_client_secret like a password field. Thoughts? > > Hmm, from

dispchar for oauth_client_secret

2025-04-15 Thread Noah Misch
commit b3f0be7 wrote: + {"oauth_scope", NULL, NULL, NULL, + "OAuth-Scope", "", 15, + offsetof(struct pg_conn, oauth_scope)}, The field containing "" is documented as follows: char *dispchar; /* Indicates how to display this field in a

Re: [18] Unintentional behavior change in commit e9931bfb75

2025-04-15 Thread Noah Misch
On Mon, Apr 14, 2025 at 12:59:50PM -0700, Jeff Davis wrote: > On Sat, 2025-04-12 at 05:34 -0700, Noah Misch wrote: > > I think the code for (2) and for "I/i in Turkish" haven't returned.  > > Given > > commit e3fa2b0 restored the v17 "I/i in Turkish"

Re: Wrong security context for deferred triggers?

2025-04-15 Thread Noah Misch
On Thu, Jan 23, 2025 at 07:28:19PM +0100, Laurenz Albe wrote: > On Thu, 2025-01-23 at 12:30 -0500, Tom Lane wrote: > > Pushed with some cosmetic adjustments > > Thank you! commit 01463e1 wrote: > +NOTICE: I am regress_groot Let's not incur trivially-avoidable trademark risks (https://google.com

Re: Call for Posters: PGConf.dev 2025

2025-04-14 Thread Noah Misch
Thanks for running this. Four logistical questions: 1. By what timestamp should one submit? 2. Where does one send the submission? On Sun, Apr 13, 2025 at 04:01:56PM +0500, Andrey Borodin wrote: > The goal of the poster session is to visually present your patch or project > on an A2-sized poste

Re: datfrozenxid > relfrozenxid w/ crash before XLOG_HEAP_INPLACE

2025-04-13 Thread Noah Misch
On Sun, Apr 06, 2025 at 11:00:54AM -0700, Noah Misch wrote: > I pushed that as commit 8e7e672 (2024-10-25). I now think DELAY_CHKPT_START > is superfluous here, per this proc.h comment: > > * (In the > * extremely common case where the data being modified is in shared buff

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2025-04-13 Thread Noah Misch
On Sat, Apr 05, 2025 at 07:09:58PM -0700, Noah Misch wrote: > On Sun, Apr 06, 2025 at 07:42:02AM +0900, Michael Paquier wrote: > > On Sat, Apr 05, 2025 at 12:13:39PM -0700, Noah Misch wrote: > > > Since the 2025-02 releases made non-toy-size archive recoveries fail > > &g

Re: Buildfarm: Enabling injection points on basilisk/dogfish (Alpine / musl)

2025-04-12 Thread Noah Misch
On Sat, Apr 12, 2025 at 07:51:06PM +0200, Wolfgang Walther wrote: > With injection points enabled, I get the following errors in test_aio: > > > [15:14:45.408](0.000s) not ok 187 - worker: first hard IO error is reported: > expected stderr > [15:14:45.409](0.000s) > [15:14:45.409](0.000s) #   Fai

Re: [18] Unintentional behavior change in commit e9931bfb75

2025-04-12 Thread Noah Misch
On Mon, Dec 02, 2024 at 10:24:07PM -0800, Jeff Davis wrote: > On Mon, 2024-12-02 at 17:25 -0500, Tom Lane wrote: > > > Should I put the special case back? > > > > I think so. > > Done. I put the special case back in (commit e3fa2b037c) because the > earlier commit wasn't intended to be a behavio

Re: type cache cleanup improvements

2025-04-11 Thread Noah Misch
On Tue, Oct 22, 2024 at 08:33:24PM +0300, Alexander Korotkov wrote: > On Tue, Oct 22, 2024 at 6:10 PM Pavel Borisov wrote: > > On Tue, 22 Oct 2024 at 11:34, Alexander Korotkov > > wrote: > >> I'm going to push this if no objections. (This became commit b85a9d0.) > + /* Call delete_rel_type

catcache search while BUFFER_LOCK_EXCLUSIVE on catalog

2025-04-10 Thread Noah Misch
4 #35 0x0093fe47 in postmaster_child_launch (child_type=B_BACKEND, child_slot=20, startup_data=0x7fff47fe66b0, startup_data_len=24, client_sock=0x7fff47fe6700) at launch_backend.c:290 #36 0x00945d55 in BackendStartup (client_sock=0x7fff47fe6700) at postmaster.c:3580 #37 0x000

Re: datfrozenxid > relfrozenxid w/ crash before XLOG_HEAP_INPLACE

2025-04-06 Thread Noah Misch
On Wed, Jun 19, 2024 at 06:29:08PM -0700, Noah Misch wrote: > https://postgr.es/m/20240512232923.aa.nmi...@google.com wrote: > > Separable, nontrivial things not fixed in the attached patch stack: > > > - Trouble is possible, I bet, if the system crashes between the

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2025-04-05 Thread Noah Misch
On Sun, Apr 06, 2025 at 07:42:02AM +0900, Michael Paquier wrote: > On Sat, Apr 05, 2025 at 12:13:39PM -0700, Noah Misch wrote: > > Since the 2025-02 releases made non-toy-size archive recoveries fail easily, > > that's not enough. If the proposed 3-second test is the wrong thi

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2025-04-05 Thread Noah Misch
On Sat, Apr 05, 2025 at 11:07:13AM -0400, Tom Lane wrote: > Michael Paquier writes: > > On Wed, Apr 02, 2025 at 05:29:00PM -0700, Noah Misch wrote: > >> Here it is. Making it fail three times took looping 1383s, 5841s, and > >> 2594s. > >> Hence, it couldn&#

Re: Back-patch of: avoid multiple hard links to same WAL file after a crash

2025-04-05 Thread Noah Misch
On Tue, Mar 11, 2025 at 06:23:15PM -0700, Noah Misch wrote: > On Wed, Mar 12, 2025 at 09:46:27AM +0900, Michael Paquier wrote: > > On Tue, Mar 11, 2025 at 01:57:49PM -0700, Noah Misch wrote: > > > Thanks for crafting back-branch versions. I've queued a task to confirm >

Re: AIO v2.5

2025-04-05 Thread Noah Misch
On Sun, Mar 23, 2025 at 11:57:48AM -0400, Andres Freund wrote: > On 2025-03-22 19:09:55 -0700, Noah Misch wrote: > > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote: > > > Attached v2.11 > > > > > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uri

Re: AIO v2.5

2025-04-05 Thread Noah Misch
On Mon, Mar 24, 2025 at 11:43:47AM -0400, Andres Freund wrote: > On 2025-03-23 17:29:39 -0700, Noah Misch wrote: > > commit 247ce06b wrote: > > > + pgaio_io_reopen(ioh); > > > + > > > + /* > > > + *

Re: AIO v2.5

2025-04-05 Thread Noah Misch
On Thu, Mar 20, 2025 at 02:54:14PM -0400, Andres Freund wrote: > On 2025-03-19 18:11:18 -0700, Noah Misch wrote: > > On Wed, Mar 19, 2025 at 06:17:37PM -0400, Andres Freund wrote: > > > On 2025-03-19 14:25:30 -0700, Noah Misch wrote: > > > > I see this relies on md_r

Re: md.c vs elog.c vs smgrreleaseall() in barrier

2025-04-05 Thread Noah Misch
On Mon, Mar 17, 2025 at 07:52:02PM -0400, Andres Freund wrote: > Here's a proposed patch for this. It turns out that the bug might already be > reachable, even without defining FDDEBUG. There's a debug ereport() in > register_dirty_segment() - but it's hard to reach in practice. > > I don't really

Re: AIO v2.5

2025-04-05 Thread Noah Misch
On Fri, Apr 04, 2025 at 11:53:13PM -0400, Andres Freund wrote: > On 2025-04-04 14:18:02 -0700, Noah Misch wrote: > > On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote: > > > > - Make DEFINED in completor before verifying page. It might be > > > > cl

Re: AIO v2.5

2025-04-04 Thread Noah Misch
On Fri, Apr 04, 2025 at 03:16:18PM -0400, Andres Freund wrote: > On 2025-04-03 12:40:23 -0700, Noah Misch wrote: > > On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote: > > In the general case, we could want client requests as follows: > > > > - If com

Re: md.c vs elog.c vs smgrreleaseall() in barrier

2025-04-04 Thread Noah Misch
ning against backpatching, but I'm not sure that's not just > laziness. It's also some risk reduction. One of these smgr APIs might have a useful interruptibility that we're now blocking. (I'm not aware of one.) > On 2025-03-19 17:45:14 -0700, Noah Misch wrote: &

Re: dblink query interruptibility

2025-04-03 Thread Noah Misch
On Wed, Mar 12, 2025 at 11:03:41AM +0100, Andreas Karlsson wrote: > On 3/12/25 12:48 AM, Noah Misch wrote: > > Overall, in the absence of objections, I will queue a task to back-patch the > > non-postgres_fdw portion of commit d3c5f37 to v13-v16. Pushed (e.g. v16 has commit 82a8

Re: AIO v2.5

2025-04-03 Thread Noah Misch
On Thu, Apr 03, 2025 at 02:19:43PM -0400, Andres Freund wrote: > 4b) > > That's not all though, after getting past this failure, I see uninitialized > memory errors for reads into temporary buffers: > > ==3334031== VALGRINDERROR-BEGIN > ==3334031== Conditional jump or move depends on uninitialise

Re: AIO v2.5

2025-04-01 Thread Noah Misch
On Tue, Apr 01, 2025 at 06:25:28PM -0400, Andres Freund wrote: > On 2025-04-01 17:47:51 -0400, Andres Freund wrote: > > 3) Some subtests fail if RELCACHE_FORCE_RELEASE and CATCACHE_FORCE_RELEASE > > are defined: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2025-04-01%2

Re: TEMP_CONFIG vs test_aio

2025-04-01 Thread Noah Misch
On Tue, Apr 01, 2025 at 03:42:44PM -0400, Andres Freund wrote: > The reason for the failure is simple, the buildfarm animal specifies > io_method=io_uring (thanks to "cookt" for setting that up so quickly, whoever > you are :)) and the test is assuming that the -c io_method=... it passes to > initd

Re: Small memory fixes for pg_createsubcriber

2025-04-01 Thread Noah Misch
On Tue, Apr 01, 2025 at 04:28:34PM -0300, Ranier Vilela wrote: > Em ter., 1 de abr. de 2025 às 15:39, Noah Misch > escreveu: > > > On Thu, Feb 27, 2025 at 10:23:31AM -0300, Ranier Vilela wrote: > > > Em qui., 27 de fev. de 2025 às 02:51, Michael Paquier < > > mi

Re: Small memory fixes for pg_createsubcriber

2025-04-01 Thread Noah Misch
On Thu, Feb 27, 2025 at 10:23:31AM -0300, Ranier Vilela wrote: > Em qui., 27 de fev. de 2025 às 02:51, Michael Paquier > escreveu: > > > On Tue, Feb 25, 2025 at 08:54:31AM -0300, Ranier Vilela wrote: > > > @@ -455,7 +455,9 @@ set_locale_and_encoding(void) > > > locale->db_locale, > > > strlen

Re: AIO v2.5

2025-04-01 Thread Noah Misch
On Tue, Apr 01, 2025 at 11:55:20AM -0400, Andres Freund wrote: > On 2025-04-01 08:11:59 -0700, Noah Misch wrote: > > On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote: > I haven't yet pushed the changes, but will work on that in the afternoon. > > I plan to

Re: AIO v2.5

2025-04-01 Thread Noah Misch
On Mon, Mar 31, 2025 at 08:41:39PM -0400, Andres Freund wrote: > updated version All non-write patches (1-7) are ready for commit, though I have some cosmetic recommendations below. I've marked the commitfest entry Ready for Committer. > + # Check a page validity error in another blo

Re: AIO v2.5

2025-03-29 Thread Noah Misch
On Sat, Mar 29, 2025 at 08:39:54PM -0400, Andres Freund wrote: > On 2025-03-29 14:29:29 -0700, Noah Misch wrote: > > On Wed, Mar 26, 2025 at 09:07:40PM -0400, Andres Freund wrote: > > The choice between LOG and LOG_SERVER_ONLY doesn't matter much for $SUBJECT. > > If a

Re: AIO v2.5

2025-03-29 Thread Noah Misch
ges - but > that's not right, the GUC hopefully is only set on a per-session basis Good catch. I've now audited the complete_shared callbacks for other variable references and actions not acceptable there. I found nothing beyond what you found by v2.14. On Sat, Mar 29, 2025 at 1

Re: AIO v2.5

2025-03-29 Thread Noah Misch
On Fri, Mar 28, 2025 at 11:35:23PM -0400, Andres Freund wrote: > The number of combinations is annoyingly large. It's e.g. plausible to use > ignore_checksum_failure=on and zero_damaged_pages=on at the same time for > recovery. That's intricate indeed. > But I finally got to a point where the cod

Re: AIO v2.5

2025-03-27 Thread Noah Misch
On Thu, Mar 27, 2025 at 04:58:11PM -0400, Andres Freund wrote: > I now wrote some tests. And I both regret doing so (because it found problems, > which would have been apparent long ago, if the feature had come with *any* > tests, if I had gone the same way I could have just pushed stuff) and am gl

Re: AIO v2.5

2025-03-27 Thread Noah Misch
On Wed, Mar 12, 2025 at 01:06:03PM -0400, Andres Freund wrote: > On 2025-03-11 20:57:43 -0700, Noah Misch wrote: > > - Like you say, "redefine max_files_per_process to be about the number of > > files each *backend* will additionally open". It will become normal that &g

Re: AIO v2.5

2025-03-26 Thread Noah Misch
On Wed, Mar 26, 2025 at 04:33:49PM -0400, Andres Freund wrote: > On 2025-03-25 17:19:15 -0700, Noah Misch wrote: > > On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote: > > Second, the aio_internal.h comment changes discussed in > > postgr.es/m/20250325155808.f7.

Re: AIO v2.5

2025-03-26 Thread Noah Misch
I reviewed everything up to and including "[PATCH v2.12 17/28] aio, bufmgr: Comment fixes", the last patch before write support. postgr.es/m/20250326001915.bc.nmi...@google.com covered patches 1-9, and this email covers patches 10-17. All remaining review comments are minor, so I've marked the com

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Tue, Mar 25, 2025 at 08:17:17PM -0400, Andres Freund wrote: > On 2025-03-25 09:15:43 -0700, Noah Misch wrote: > > On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote: > > > FWIW, I prototyped this, it's not hard. > > > > > > But it can't

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Mon, Mar 24, 2025 at 09:18:06PM -0400, Andres Freund wrote: > Attached v2.12, with the following changes: > TODO: > Wonder if it's worth adding some coverage for when checksums are disabled? > Probably not necessary? Probably not necessary, agreed. Orthogonal to AIO, it's likely worth a

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Tue, Mar 25, 2025 at 04:56:53PM -0400, Andres Freund wrote: > The repeated-iteration approach taken in pgaio_closing_fd() isn't the > prettiest, but it's hard to to imagine that ever being a noticeable. Yep. I've reviewed the fixup code, and it looks all good. > This survives a testrun where

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Tue, Mar 25, 2025 at 04:07:35PM -0400, Andres Freund wrote: > On 2025-03-25 12:39:56 -0700, Noah Misch wrote: > > On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote: > > > There are 2 1/2 ways around this: > > > > > > 1) Stop using IOSQE_ASYNC

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Tue, Mar 25, 2025 at 02:58:37PM -0400, Andres Freund wrote: > On 2025-03-25 08:58:08 -0700, Noah Misch wrote: > > While having nagging thoughts that we might be releasing FDs before io_uring > > gets them into kernel custody, I tried this hack to maximize FD turnover: >

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Tue, Mar 25, 2025 at 11:57:58AM -0400, Andres Freund wrote: > On 2025-03-25 07:11:20 -0700, Noah Misch wrote: > > On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote: > > > If we want to implement it, I think we could introduce PGAIO_RS_WARN, > > > whic

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Tue, Mar 25, 2025 at 11:26:14AM -0400, Andres Freund wrote: > On 2025-03-25 06:33:21 -0700, Noah Misch wrote: > > On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote: > > > On 2025-03-24 17:45:37 -0700, Noah Misch wrote: > > > > (We may be due for a test

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Mon, Mar 24, 2025 at 10:52:19PM -0400, Andres Freund wrote: > On 2025-03-24 19:20:37 -0700, Noah Misch wrote: > > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote: > > > +static pg_attribute_always_inline PgAioResult > > > +buffer_readv_complete_one(u

Re: AIO v2.5

2025-03-25 Thread Noah Misch
On Mon, Mar 24, 2025 at 10:30:27PM -0400, Andres Freund wrote: > On 2025-03-24 17:45:37 -0700, Noah Misch wrote: > > (We may be due for a test mode that does smgrreleaseall() at every > > CHECK_FOR_INTERRUPTS()?) > > I suspect we are. I'm a bit afraid of even tr

Re: AIO v2.5

2025-03-24 Thread Noah Misch
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote: > Subject: [PATCH v2.11 09/27] bufmgr: Implement AIO read support [I checked that v2.12 doesn't invalidate these review comments, but I didn't technically rebase the review onto v2.12's line numbers.] > static void > TerminateBuffer

Re: AIO v2.5

2025-03-23 Thread Noah Misch
On Sun, Mar 23, 2025 at 11:11:53AM -0400, Andres Freund wrote: > On 2025-03-22 17:20:56 -0700, Noah Misch wrote: > > On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote: > > > Not sure yet how to best disable testing io_uring in this case. We can't > > >

Re: AIO v2.5

2025-03-23 Thread Noah Misch
commit 247ce06b wrote: > + pgaio_io_reopen(ioh); > + > + /* > + * To be able to exercise the reopen-fails path, allow > injection > + * points to trigger a failure at this point. > + */ > +

Re: AIO v2.5

2025-03-22 Thread Noah Misch
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote: > Attached v2.11, with the following changes: > - Added an error check for FileStartReadV() failing > > FileStartReadV() actually can fail, if the file can't be re-opened. I > thought it'd be important for the error message to dif

Re: AIO v2.5

2025-03-22 Thread Noah Misch
On Thu, Mar 20, 2025 at 09:58:37PM -0400, Andres Freund wrote: > Attached v2.11 > Subject: [PATCH v2.11 05/27] aio: Add io_method=io_uring Apart from some isolated cosmetic points, this is ready to commit: > + ereport(ERROR, > + errcode(err

Re: AIO v2.5

2025-03-20 Thread Noah Misch
On Thu, Mar 20, 2025 at 01:05:05PM -0400, Andres Freund wrote: > On 2025-03-19 18:17:37 -0400, Andres Freund wrote: > > On 2025-03-19 14:25:30 -0700, Noah Misch wrote: > > > > +* marked as failed. In case of a partial read, some > > > > buffers may

  1   2   3   4   5   6   7   8   9   10   >