pgsql: jit: Support opaque pointers in LLVM 16.

2023-10-18 Thread Thomas Munro
jit: Support opaque pointers in LLVM 16. Remove use of LLVMGetElementType() and provide the type of all pointers to LLVMBuildXXX() functions when emitting IR, as required by modern LLVM versions[1]. * For LLVM <= 14, we'll still use the old LLVMBuildXXX() functions. * For LLVM == 15, we'll

Re: Add support for AT LOCAL

2023-10-17 Thread Thomas Munro
On Wed, Oct 18, 2023 at 11:54 AM Tom Lane wrote: > Thomas Munro writes: > > > Given that IBM describes xlc as "legacy" (replaced by xlclang, but > > still supported for some unspecified period of time for the benefit of > > people who need C++ ABI compatib

Re: Add support for AT LOCAL

2023-10-17 Thread Thomas Munro
Hmm, I guess I must have missed some important flag or environment variable when trying to reproduce it, sorry. Given that IBM describes xlc as "legacy" (replaced by xlclang, but still supported for some unspecified period of time for the benefit of people who need C++ ABI compatibility with old

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-10-16 Thread Thomas Munro
these (two birds with one stone), so I'll just leave these here to keep the cfbot happy in the meantime. From 3ab478ba3bb1712e6fc529f4bdaf48b58a6d72c5 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 26 Jul 2023 08:46:19 +1200 Subject: [PATCH v7 1/2] Update control file atomically during backups

pgsql: Try to handle torn reads of pg_control in frontend.

2023-10-15 Thread Thomas Munro
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by

pgsql: Try to handle torn reads of pg_control in frontend.

2023-10-15 Thread Thomas Munro
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by

pgsql: Try to handle torn reads of pg_control in frontend.

2023-10-15 Thread Thomas Munro
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by

pgsql: Try to handle torn reads of pg_control in frontend.

2023-10-15 Thread Thomas Munro
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by

pgsql: Try to handle torn reads of pg_control in frontend.

2023-10-15 Thread Thomas Munro
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by

pgsql: Try to handle torn reads of pg_control in frontend.

2023-10-15 Thread Thomas Munro
Try to handle torn reads of pg_control in frontend. Some of our src/bin tools read the control file without any kind of interlocking against concurrent writes from the server. At least ext4 and ntfs can expose partially modified contents when you do that. For now, we'll try to tolerate this by

Re: Add support for AT LOCAL

2023-10-15 Thread Thomas Munro
On Mon, Oct 16, 2023 at 4:02 PM Tom Lane wrote: > I'm having a hard time not believing that this is a compiler bug. > Looking back at 8d2a01ae12cd and its speculation that xlc is overly > liberal about reordering code around sequence points ... I wonder > if it'd help to do this calculation in a

Re: Add support for AT LOCAL

2023-10-15 Thread Thomas Munro
On Mon, Oct 16, 2023 at 2:58 PM Michael Paquier wrote: > Another theory would be one of these weird compiler optimization issue > from xlc? In recent history, there was 8d2a01ae12cd. Yeah, there are more like that too. xlc 12.1 is dead (like the OS version it shipped with). New versions are

Re: pgsql: Rename 005_login_trigger.pl to 006_login_trigger.pl

2023-10-15 Thread Thomas Munro
On Mon, Oct 16, 2023 at 2:04 PM Alexander Korotkov wrote: > Rename 005_login_trigger.pl to 006_login_trigger.pl > > In order to avoid numbering collision with 005_sspi.pl. Hi Alexander, Usually I notice if someone steals my TAP script number because there is git conflict in eg

pgsql: Fix comment from commit 22655aa231.

2023-10-15 Thread Thomas Munro
Fix comment from commit 22655aa231. Per automated complaint from BF animal koel this needed to be re-indented, but there was also a typo. Back-patch to 16. Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/3090213690bd69dd46dae865a2021982e7309208 Modified

pgsql: Fix comment from commit 22655aa231.

2023-10-15 Thread Thomas Munro
Fix comment from commit 22655aa231. Per automated complaint from BF animal koel this needed to be re-indented, but there was also a typo. Back-patch to 16. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/01529c7040088db2718628d0814058598152bd39 Modified Files

Re: Add support for AT LOCAL

2023-10-15 Thread Thomas Munro
On Mon, Oct 16, 2023 at 11:24 AM Thomas Munro wrote: > On Mon, Oct 16, 2023 at 10:57 AM Tom Lane wrote: > > I'm tempted to wonder if this helps: > > > > - result->time = t->time + (t->zone - tz) * USECS_PER_SEC; > > + result->time = t->tim

Re: Add support for AT LOCAL

2023-10-15 Thread Thomas Munro
On Mon, Oct 16, 2023 at 10:57 AM Tom Lane wrote: > I'm tempted to wonder if this helps: > > - result->time = t->time + (t->zone - tz) * USECS_PER_SEC; > + result->time = t->time + (int64) (t->zone - tz) * USECS_PER_SEC; I wanted to be able to try this and any other theories and

pgsql: Acquire ControlFileLock in relevant SQL functions.

2023-10-15 Thread Thomas Munro
Acquire ControlFileLock in relevant SQL functions. Commit dc7d70ea added functions that read the control file, but didn't acquire ControlFileLock. With unlucky timing, file systems that have weak interlocking like ext4 and ntfs could expose partially overwritten contents, and the checksum would

pgsql: Acquire ControlFileLock in relevant SQL functions.

2023-10-15 Thread Thomas Munro
Acquire ControlFileLock in relevant SQL functions. Commit dc7d70ea added functions that read the control file, but didn't acquire ControlFileLock. With unlucky timing, file systems that have weak interlocking like ext4 and ntfs could expose partially overwritten contents, and the checksum would

pgsql: Acquire ControlFileLock in relevant SQL functions.

2023-10-15 Thread Thomas Munro
Acquire ControlFileLock in relevant SQL functions. Commit dc7d70ea added functions that read the control file, but didn't acquire ControlFileLock. With unlucky timing, file systems that have weak interlocking like ext4 and ntfs could expose partially overwritten contents, and the checksum would

pgsql: Acquire ControlFileLock in relevant SQL functions.

2023-10-15 Thread Thomas Munro
Acquire ControlFileLock in relevant SQL functions. Commit dc7d70ea added functions that read the control file, but didn't acquire ControlFileLock. With unlucky timing, file systems that have weak interlocking like ext4 and ntfs could expose partially overwritten contents, and the checksum would

pgsql: Acquire ControlFileLock in relevant SQL functions.

2023-10-15 Thread Thomas Munro
Acquire ControlFileLock in relevant SQL functions. Commit dc7d70ea added functions that read the control file, but didn't acquire ControlFileLock. With unlucky timing, file systems that have weak interlocking like ext4 and ntfs could expose partially overwritten contents, and the checksum would

pgsql: Acquire ControlFileLock in relevant SQL functions.

2023-10-15 Thread Thomas Munro
Acquire ControlFileLock in relevant SQL functions. Commit dc7d70ea added functions that read the control file, but didn't acquire ControlFileLock. With unlucky timing, file systems that have weak interlocking like ext4 and ntfs could expose partially overwritten contents, and the checksum would

pgsql: Acquire ControlFileLock in relevant SQL functions.

2023-10-15 Thread Thomas Munro
Acquire ControlFileLock in relevant SQL functions. Commit dc7d70ea added functions that read the control file, but didn't acquire ControlFileLock. With unlucky timing, file systems that have weak interlocking like ext4 and ntfs could expose partially overwritten contents, and the checksum would

Re: Add support for AT LOCAL

2023-10-15 Thread Thomas Munro
One of the AIX animals gave a strange result here: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet=2023-10-15%2011%3A40%3A01 If you ignore the diffs due to change in column width, the interesting change seems to be: - 23:59:00-07| 06:59:00+00| 06:59:00+00|

Re: LLVM 16 (opaque pointers)

2023-10-13 Thread Thomas Munro
On Sat, Oct 14, 2023 at 3:56 AM Andres Freund wrote: > On 2023-10-13 16:44:13 +0200, Dmitry Dolgov wrote: > > Here is what I had in mind (only this part in the second patch was changed). > > Makes sense to me. I think we'll likely eventually want to use a custom > pipeline anyway, and I think we

Re: LLVM 16 (opaque pointers)

2023-10-13 Thread Thomas Munro
On Wed, Oct 11, 2023 at 10:31 PM Ronan Dunklau wrote: > Le mercredi 11 octobre 2023, 10:59:50 CEST Thomas Munro a écrit : > > The back-patch to 12 was a little trickier than anticipated, but after > > taking a break and trying again I now have PG 12...17 patches that > > I'v

Re: Wait events for delayed checkpoints

2023-10-12 Thread Thomas Munro
On Fri, Oct 13, 2023 at 2:19 PM Robert Haas wrote: > On Thu, Oct 12, 2023 at 7:09 PM Michael Paquier wrote: > > HaveVirtualXIDsDelayingChkpt() does immediately a LWLockAcquire() > > which would itself report a wait event for ProcArrayLock, overwriting > > this new one, no? > > Ah, right: the

pgsql: Add wait events for checkpoint delay mechanism.

2023-10-12 Thread Thomas Munro
Add wait events for checkpoint delay mechanism. When MyProc->delayChkptFlags is set to temporarily block phase transitions in a concurrent checkpoint, the checkpointer enters a sleep-poll loop to wait for the flag to be cleared. We should show that as a wait event in the pg_stat_activity view.

Wait events for delayed checkpoints

2023-10-11 Thread Thomas Munro
e other side as cheap as possible, so we don't care about making the checkpointer take some micro-naps in this case. I feel like the key point here is that it's waiting for another process to do stuff and unblock it. From fdce1ce74af59efa9020eecf52fe52af07b96670 Mon Sep 17 00:00:00 2001 From: Thoma

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-10-11 Thread Thomas Munro
I'm planning to push 0002 (retries in frontend programs, which is where this thread began) and 0004 (add missing locks to SQL functions), including back-patches as far as 12, in a day or so. I'll abandon the others for now, since we're now thinking bigger[1] for backups, side stepping the

Re: The danger of deleting backup_label

2023-10-11 Thread Thomas Munro
Hi David, Even though I spent a whole bunch of time trying to figure out how to make concurrent reads of the control file sufficiently atomic for backups (pg_basebackup and low level filesystem tools), and we explored multiple avenues with varying results, and finally came up with something that

Re: New WAL record to detect the checkpoint redo location

2023-10-11 Thread Thomas Munro
On Tue, Oct 10, 2023 at 11:33 AM Robert Haas wrote: > On Mon, Oct 9, 2023 at 4:47 PM Andres Freund wrote: > > I think we might be able to speed some of this up by pre-compute values so > > we > > can implement things like bytesleft / UsableBytesInPage with shifts. IIRC we > > already insist on

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Thomas Munro
On Thu, Oct 12, 2023 at 9:27 AM Thomas Munro wrote: > On Thu, Oct 12, 2023 at 9:05 AM Robert Haas wrote: > > But if we do want to keep those cross-checks, why not take what Thomas > > proposed a little further and move all of xlp_sysid, xlp_seg_size, and >

Re: Lowering the default wal_blocksize to 4K

2023-10-11 Thread Thomas Munro
On Thu, Oct 12, 2023 at 9:05 AM Robert Haas wrote: > But if we do want to keep those cross-checks, why not take what Thomas > proposed a little further and move all of xlp_sysid, xlp_seg_size, and > xlp_xlog_blcksz into XLOG_CHECKPOINT_REDO? Then long and short page > headers would become

Re: LLVM 16 (opaque pointers)

2023-10-11 Thread Thomas Munro
On Thu, Sep 21, 2023 at 12:47 PM Thomas Munro wrote: > On Thu, Sep 21, 2023 at 12:24 PM Devrim Gündüz wrote: > > RHEL releases new LLVM version along with their new minor releases every > > 6 month, and we have to build older versions with new LLVM each time. > > Fr

Re: Lowering the default wal_blocksize to 4K

2023-10-10 Thread Thomas Munro
On Wed, Oct 11, 2023 at 12:29 PM Andres Freund wrote: > On 2023-10-10 21:30:44 +0200, Matthias van de Meent wrote: > > On Tue, 10 Oct 2023 at 06:14, Andres Freund wrote: > > > I was thinking we should perhaps do the opposite, namely getting rid of > > > short > > > page headers. The overhead in

Re: [PATCH] hstore: Fix parsing on Mac OS X: isspace() is locale specific

2023-10-09 Thread Thomas Munro
FTR I ran into a benign case of the phenomenon in this thread when dealing with row types. In rowtypes.c, we double-quote stuff containing spaces, but we detect them by passing individual bytes of UTF-8 sequences to isspace(). Like macOS, Windows thinks that 0xa0 is a space when you do that, so

Re: Proposal to use JSON for Postgres Parser format

2023-10-09 Thread Thomas Munro
On Tue, Sep 20, 2022 at 4:16 PM Thomas Munro wrote: > To explain my earlier guess: reader code for #S(STRUCTNAME ...) can > bee seen here, though it's being lexed as "PLAN_SYM" so perhaps the > author of that C already didn't know that was a general syntax for > Lisp structs

Re: REL_15_STABLE: pgbench tests randomly failing on CI, Windows only

2023-10-08 Thread Thomas Munro
On Mon, Oct 9, 2023 at 3:25 PM Noah Misch wrote: > The "fd >= FD_SETSIZE" check is irrelevant on Windows. See comments in the > attached patch; in brief, Windows assigns FDs and uses FD_SETSIZE differently. > The first associated failure was commit dea12a1 (2023-08-03); as a doc commit, > it's

Re: CREATE DATABASE with filesystem cloning

2023-10-08 Thread Thomas Munro
xtents (contiguous ranges of physical blocks to be referenced by the new file). [1] https://lore.kernel.org/linux-btrfs/ae81e48b0e954bae1c3451c0da1a24ae7146606c.1676684984.git.bo...@bur.io/T/#u From dd5c07d873e90a6feac371d2879015a5e6154632 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 2 Sep 2023

Re: pg_upgrade --copy-file-range

2023-10-07 Thread Thomas Munro
handle that in > parseCommandLine(). I suggest following the clone example for these, > since the issues there are very similar. Done. From 9ea1c3fc39a47f634a4fffd1ff1c9b9cf0299d65 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 2 Jun 2023 13:35:54 -0400 Subject: [PATCH v2] Add --copy

Re: CREATE DATABASE with filesystem cloning

2023-10-07 Thread Thomas Munro
One thing I forgot to mention about this experiment: when used in a backend I think this probably needs a chunk size (what size?) and a CFI(). Which is a bit annoying, because for best results we want SSIZE_MAX, but in the case that copy_file_range() falls back to raw copy, that'd do I/O work

CREATE DATABASE with filesystem cloning

2023-10-06 Thread Thomas Munro
00:00:00 2001 From: Thomas Munro Date: Sat, 2 Sep 2023 22:21:49 +1200 Subject: [PATCH] WIP: CREATE DATABASE ... STRATEGY=FILE_CLONE. Similar to STRATEGY=FILE_COPY, but using facilities that tell the OS explicitly that we're copying, so that it has the opportunity to share block ranges in copy

Re: Checks in RegisterBackgroundWorker.()

2023-10-06 Thread Thomas Munro
On Thu, Sep 28, 2023 at 9:46 AM Heikki Linnakangas wrote: > Here's a new version of these patches. I fixed one comment and ran > pgindent, no other changes. > Subject: [PATCH v2 1/3] Clarify the checks in RegisterBackgroundWorker. LGTM. I see it passes on CI, and I also tested locally with

Re: dikkop seems unhappy because of openssl stuff (FreeBSD 14-BETA1)

2023-09-29 Thread Thomas Munro
Does the image lack a /etc/localtime file/link, but perhaps one of you did something to create it? This came up with the CI image: https://www.postgresql.org/message-id/flat/20230731191510.pebqeiuo2sbmlcfh%40awork3.anarazel.de Also mentioned at: https://wiki.tcl-lang.org/page/clock+scan

Re: how to manage Cirrus on personal repository

2023-09-29 Thread Thomas Munro
On Sat, Sep 30, 2023 at 3:35 AM Nazir Bilal Yavuz wrote: > On Fri, 29 Sept 2023 at 13:24, Peter Eisentraut wrote: > > I have a private repository on GitHub where I park PostgreSQL > > development work. I also have Cirrus activated on that repository, to > > build those branches, using the

Re: Streaming I/O, vectored I/O (WIP)

2023-09-27 Thread Thomas Munro
On Thu, Sep 28, 2023 at 9:13 AM Andres Freund wrote: > On 2023-09-27 21:33:15 +0300, Heikki Linnakangas wrote: > > Looking at the later patch that introduces the streaming read API, seems > > that it finishes all the reads, so I suppose we don't need an abort > > function. Does it all get cleaned

Re: Fail hard if xlogreader.c fails on out-of-memory

2023-09-26 Thread Thomas Munro
On Tue, Sep 26, 2023 at 8:38 PM Michael Paquier wrote: > Thoughts and/or comments are welcome. I don't have an opinion yet on your other thread about making this stuff configurable for replicas, but for the simple crash recovery case shown here, hard failure makes sense to me. Here are some

pgsql: Fix edge-case for xl_tot_len broken by bae868ca.

2023-09-25 Thread Thomas Munro
Fix edge-case for xl_tot_len broken by bae868ca. bae868ca removed a check that was still needed. If you had an xl_tot_len at the end of a page that was too small for a record header, but not big enough to span onto the next page, we'd immediately perform the CRC check using a bogus large length.

pgsql: Fix edge-case for xl_tot_len broken by bae868ca.

2023-09-25 Thread Thomas Munro
Fix edge-case for xl_tot_len broken by bae868ca. bae868ca removed a check that was still needed. If you had an xl_tot_len at the end of a page that was too small for a record header, but not big enough to span onto the next page, we'd immediately perform the CRC check using a bogus large length.

pgsql: Fix edge-case for xl_tot_len broken by bae868ca.

2023-09-25 Thread Thomas Munro
Fix edge-case for xl_tot_len broken by bae868ca. bae868ca removed a check that was still needed. If you had an xl_tot_len at the end of a page that was too small for a record header, but not big enough to span onto the next page, we'd immediately perform the CRC check using a bogus large length.

pgsql: Fix edge-case for xl_tot_len broken by bae868ca.

2023-09-25 Thread Thomas Munro
Fix edge-case for xl_tot_len broken by bae868ca. bae868ca removed a check that was still needed. If you had an xl_tot_len at the end of a page that was too small for a record header, but not big enough to span onto the next page, we'd immediately perform the CRC check using a bogus large length.

pgsql: Fix edge-case for xl_tot_len broken by bae868ca.

2023-09-25 Thread Thomas Munro
Fix edge-case for xl_tot_len broken by bae868ca. bae868ca removed a check that was still needed. If you had an xl_tot_len at the end of a page that was too small for a record header, but not big enough to span onto the next page, we'd immediately perform the CRC check using a bogus large length.

pgsql: Fix edge-case for xl_tot_len broken by bae868ca.

2023-09-25 Thread Thomas Munro
Fix edge-case for xl_tot_len broken by bae868ca. bae868ca removed a check that was still needed. If you had an xl_tot_len at the end of a page that was too small for a record header, but not big enough to span onto the next page, we'd immediately perform the CRC check using a bogus large length.

pgsql: Don't use Perl pack('Q') in 039_end_of_wal.pl.

2023-09-22 Thread Thomas Munro
Don't use Perl pack('Q') in 039_end_of_wal.pl. 'Q' for 64 bit integers turns out not to work on 32 bit Perl, as revealed by the build farm. Use 'II' instead, and deal with endianness. Back-patch to 12, like bae868ca. Discussion: https://postgr.es/m/ZQ4r1vHcryBsSi_V%40paquier.xyz Branch --

pgsql: Don't use Perl pack('Q') in 039_end_of_wal.pl.

2023-09-22 Thread Thomas Munro
Don't use Perl pack('Q') in 039_end_of_wal.pl. 'Q' for 64 bit integers turns out not to work on 32 bit Perl, as revealed by the build farm. Use 'II' instead, and deal with endianness. Back-patch to 12, like bae868ca. Discussion: https://postgr.es/m/ZQ4r1vHcryBsSi_V%40paquier.xyz Branch --

pgsql: Don't use Perl pack('Q') in 039_end_of_wal.pl.

2023-09-22 Thread Thomas Munro
Don't use Perl pack('Q') in 039_end_of_wal.pl. 'Q' for 64 bit integers turns out not to work on 32 bit Perl, as revealed by the build farm. Use 'II' instead, and deal with endianness. Back-patch to 12, like bae868ca. Discussion: https://postgr.es/m/ZQ4r1vHcryBsSi_V%40paquier.xyz Branch --

pgsql: Don't use Perl pack('Q') in 039_end_of_wal.pl.

2023-09-22 Thread Thomas Munro
Don't use Perl pack('Q') in 039_end_of_wal.pl. 'Q' for 64 bit integers turns out not to work on 32 bit Perl, as revealed by the build farm. Use 'II' instead, and deal with endianness. Back-patch to 12, like bae868ca. Discussion: https://postgr.es/m/ZQ4r1vHcryBsSi_V%40paquier.xyz Branch --

pgsql: Don't use Perl pack('Q') in 039_end_of_wal.pl.

2023-09-22 Thread Thomas Munro
Don't use Perl pack('Q') in 039_end_of_wal.pl. 'Q' for 64 bit integers turns out not to work on 32 bit Perl, as revealed by the build farm. Use 'II' instead, and deal with endianness. Back-patch to 12, like bae868ca. Discussion: https://postgr.es/m/ZQ4r1vHcryBsSi_V%40paquier.xyz Branch --

pgsql: Don't use Perl pack('Q') in 039_end_of_wal.pl.

2023-09-22 Thread Thomas Munro
Don't use Perl pack('Q') in 039_end_of_wal.pl. 'Q' for 64 bit integers turns out not to work on 32 bit Perl, as revealed by the build farm. Use 'II' instead, and deal with endianness. Back-patch to 12, like bae868ca. Discussion: https://postgr.es/m/ZQ4r1vHcryBsSi_V%40paquier.xyz Branch --

Failures on gombessa -- EIO?

2023-09-22 Thread Thomas Munro
I am not sure why REL_16_STABLE fails consistently as of ~4 days ago. It seems like bad storage or something? Just now it happened also on HEAD. I wonder why it would be sensitive to the branch.

pgsql: Don't trust unvalidated xl_tot_len.

2023-09-22 Thread Thomas Munro
ing, but changing that is left for later work. Also included is a new TAP test to exercise various cases of end-of-WAL detection by writing contrived data into the WAL from Perl. Back-patch to 12. We decided not to put this change into the final release of 11. Author: Thomas Munro Author: Mich

pgsql: Don't trust unvalidated xl_tot_len.

2023-09-22 Thread Thomas Munro
ing, but changing that is left for later work. Also included is a new TAP test to exercise various cases of end-of-WAL detection by writing contrived data into the WAL from Perl. Back-patch to 12. We decided not to put this change into the final release of 11. Author: Thomas Munro Author: Mich

pgsql: Don't trust unvalidated xl_tot_len.

2023-09-22 Thread Thomas Munro
ing, but changing that is left for later work. Also included is a new TAP test to exercise various cases of end-of-WAL detection by writing contrived data into the WAL from Perl. Back-patch to 12. We decided not to put this change into the final release of 11. Author: Thomas Munro Author: Mich

pgsql: Don't trust unvalidated xl_tot_len.

2023-09-22 Thread Thomas Munro
ing, but changing that is left for later work. Also included is a new TAP test to exercise various cases of end-of-WAL detection by writing contrived data into the WAL from Perl. Back-patch to 12. We decided not to put this change into the final release of 11. Author: Thomas Munro Author: Mich

pgsql: Don't trust unvalidated xl_tot_len.

2023-09-22 Thread Thomas Munro
ing, but changing that is left for later work. Also included is a new TAP test to exercise various cases of end-of-WAL detection by writing contrived data into the WAL from Perl. Back-patch to 12. We decided not to put this change into the final release of 11. Author: Thomas Munro Author: Mich

pgsql: Don't trust unvalidated xl_tot_len.

2023-09-22 Thread Thomas Munro
ing, but changing that is left for later work. Also included is a new TAP test to exercise various cases of end-of-WAL detection by writing contrived data into the WAL from Perl. Back-patch to 12. We decided not to put this change into the final release of 11. Author: Thomas Munro Author: Mich

Re: LLVM 16 (opaque pointers)

2023-09-20 Thread Thomas Munro
On Thu, Sep 21, 2023 at 12:24 PM Devrim Gündüz wrote: > On Thu, 2023-09-21 at 08:22 +1200, Thomas Munro wrote: > > So far I've tested LLVM versions 10, 15, 16, 17, 18 (= their main > > branch) against PostgreSQL versions 14, 15, 16. I've attached the > > versions that a

Guiding principle for dropping LLVM versions?

2023-09-20 Thread Thomas Munro
Hi, Currently we claim to support all versions of LLVM from 3.9 up. It's now getting quite inconvenient to test changes on older releases with single digit major versions, because they aren't available through usual package channels on current distributions, and frankly it feels like pointless

Re: LLVM 16 (opaque pointers)

2023-09-20 Thread Thomas Munro
this version I changed it to what I hope is a more obvious/explicit expression of that goal: + AssertVariableIsOfType(, + ExecEvalSubroutine); [1] https://github.com/macdice/postgres/tree/llvm16-14 and -15 From d5e4ac6dd9c2016c00bedfd8cd404e2f277701e1 Mon Sep

Re: dikkop seems unhappy because of openssl stuff (FreeBSD 14-BETA1)

2023-09-18 Thread Thomas Munro
On Tue, Sep 19, 2023 at 2:04 PM Michael Paquier wrote: > On Mon, Sep 18, 2023 at 03:11:27PM +0200, Tomas Vondra wrote: > > Both 11 and 12 failed with a weird openssl segfaults in plpython tests, > > see [2] and [3]. And 13 is stuck in some openssl stuff in plpython > > tests, with 100% CPU usage

Re: dikkop seems unhappy because of openssl stuff (FreeBSD 14-BETA1)

2023-09-18 Thread Thomas Munro
IDK, but I tried installing tcl87 as you showed in packages.txt, and REL_11_STABLE said: checking for tclsh... no checking for tcl... no checking for tclsh8.6... no checking for tclsh86... no checking for tclsh8.5... no checking for tclsh85... no checking for tclsh8.4... no checking for

Re: pgsql 10.23 , different systems, same table , same plan, different Buffers: shared hit

2023-09-15 Thread Thomas Munro
On Sat, Sep 16, 2023 at 7:42 AM Tom Lane wrote: > Sadly, this proves very little about Linux's behavior. glibc's idea > of en_US involves some very complicated multi-pass sort rules. > AFAICT from the FreeBSD sort(1) man page, FreeBSD defines en_US > as "same as C except case-insensitive",

Re: Cygwin cleanup

2023-09-14 Thread Thomas Munro
On Wed, Feb 8, 2023 at 8:06 PM Thomas Munro wrote: > On Fri, Jan 13, 2023 at 5:17 PM Justin Pryzby wrote: > > My patch used fsync_fname_ext() which would cause an ERROR rather than a > > PANIC when failing to fsync the logical decoding pathname. > > FTR While analysing a

Re: [PATCH] Add CANONICAL option to xmlserialize

2023-09-14 Thread Thomas Munro
On Thu, Sep 14, 2023 at 11:54 PM Jim Jones wrote: > The cfbot started complaining about this patch on "macOS - Ventura - Meson" > > 'Persistent worker failed to start the task: tart isolation failed: failed to > create VM cloned from "ghcr.io/cirruslabs/macos-ventura-base:latest": tart >

pgsql: Fix exception safety bug in typcache.c.

2023-09-12 Thread Thomas Munro
Fix exception safety bug in typcache.c. If an out-of-memory error was thrown at an unfortunate time, ensure_record_cache_typmod_slot_exists() could leak memory and leave behind a global state that produced an infinite loop on the next call. Fix by merging RecordCacheArray and

pgsql: Fix exception safety bug in typcache.c.

2023-09-12 Thread Thomas Munro
Fix exception safety bug in typcache.c. If an out-of-memory error was thrown at an unfortunate time, ensure_record_cache_typmod_slot_exists() could leak memory and leave behind a global state that produced an infinite loop on the next call. Fix by merging RecordCacheArray and

pgsql: Fix exception safety bug in typcache.c.

2023-09-12 Thread Thomas Munro
Fix exception safety bug in typcache.c. If an out-of-memory error was thrown at an unfortunate time, ensure_record_cache_typmod_slot_exists() could leak memory and leave behind a global state that produced an infinite loop on the next call. Fix by merging RecordCacheArray and

pgsql: Fix exception safety bug in typcache.c.

2023-09-12 Thread Thomas Munro
Fix exception safety bug in typcache.c. If an out-of-memory error was thrown at an unfortunate time, ensure_record_cache_typmod_slot_exists() could leak memory and leave behind a global state that produced an infinite loop on the next call. Fix by merging RecordCacheArray and

pgsql: Fix exception safety bug in typcache.c.

2023-09-12 Thread Thomas Munro
Fix exception safety bug in typcache.c. If an out-of-memory error was thrown at an unfortunate time, ensure_record_cache_typmod_slot_exists() could leak memory and leave behind a global state that produced an infinite loop on the next call. Fix by merging RecordCacheArray and

pgsql: Fix exception safety bug in typcache.c.

2023-09-12 Thread Thomas Munro
Fix exception safety bug in typcache.c. If an out-of-memory error was thrown at an unfortunate time, ensure_record_cache_typmod_slot_exists() could leak memory and leave behind a global state that produced an infinite loop on the next call. Fix by merging RecordCacheArray and

pgsql: Fix exception safety bug in typcache.c.

2023-09-12 Thread Thomas Munro
Fix exception safety bug in typcache.c. If an out-of-memory error was thrown at an unfortunate time, ensure_record_cache_typmod_slot_exists() could leak memory and leave behind a global state that produced an infinite loop on the next call. Fix by merging RecordCacheArray and

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-09-11 Thread Thomas Munro
On Sat, Sep 9, 2023 at 9:00 PM Alexander Lakhin wrote: > Yes, I think we deal with something like that. I can try to deduce a minimum > change that affects reproducing the issue, but may be it's not that important. > Perhaps we now should think of escalating the problem to FreeBSD developers? > I

Re: Query execution in Perl TAP tests needs work

2023-09-11 Thread Thomas Munro
On Sun, Sep 3, 2023 at 12:17 PM Thomas Munro wrote: > (Huh, while contemplating trying that, I just noticed that the GCC > build farm's AIX 7.2 system seems to have given up the ghost a few > weeks ago. I wonder if it'll come back online with the current > release, or if th

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-09-08 Thread Thomas Munro
On Sat, Sep 9, 2023 at 7:00 AM Alexander Lakhin wrote: > It takes less than 10 minutes on average for me. I checked > REL_12_STABLE, REL_13_STABLE, and REL_14_STABLE (with HAVE_KQUEUE undefined > forcefully) — they all are affected. > I could not reproduce the lockup on my Ubuntu box (with

Re: WL_SOCKET_ACCEPT fairness on Windows

2023-09-08 Thread Thomas Munro
I committed this for 17. It would be good to come up with something fundamentally better than this, to get rid of that 64 event limit nonsense, but I don't see it happening in the 17 cycle, and prefer the semantics with this commit in the meantime.

pgsql: Teach WaitEventSetWait() to report multiple events on Windows.

2023-09-08 Thread Thomas Munro
Teach WaitEventSetWait() to report multiple events on Windows. The WAIT_USE_WIN32 implementation of WaitEventSetWait() previously reported at most one event per call, because that's what the underlying WaitForMultipleObjects() call does. We can make the behavior match the three Unix

Re: old_snapshot_threshold bottleneck on replica

2023-09-07 Thread Thomas Munro
On Fri, Sep 8, 2023 at 4:22 PM Peter Geoghegan wrote: > This looks right to me. Thanks, pushed.

pgsql: Remove some more "snapshot too old" vestiges.

2023-09-07 Thread Thomas Munro
Remove some more "snapshot too old" vestiges. Commit f691f5b8 removed the logic, but left behind some now-useless Snapshot arguments to various AM-internal functions, and missed a couple of comments. Reported-by: Peter Geoghegan Discussion:

Re: old_snapshot_threshold bottleneck on replica

2023-09-07 Thread Thomas Munro
On Fri, Sep 8, 2023 at 2:00 PM Thomas Munro wrote: > On Fri, Sep 8, 2023 at 1:53 PM Peter Geoghegan wrote: > > Thanks for working on this. Though I wonder why you didn't do > > something closer to a straight revert of the feature. Why is nbtree > > still passing around

Re: old_snapshot_threshold bottleneck on replica

2023-09-07 Thread Thomas Munro
On Fri, Sep 8, 2023 at 1:53 PM Peter Geoghegan wrote: > On Tue, Sep 5, 2023 at 12:58 AM Thomas Munro wrote: > > I hope we get "snapshot too old" back one day. > > Thanks for working on this. Though I wonder why you didn't do > something closer to a straight revert o

Re: Cutting support for OpenSSL 1.0.1 and 1.0.2 in 17~?

2023-09-07 Thread Thomas Munro
On Fri, Sep 8, 2023 at 11:48 AM Michael Paquier wrote: > On Thu, Sep 07, 2023 at 01:44:11PM +0200, Daniel Gustafsson wrote: > > Sadly I wouldn't be the least bit surprised if there are 1.0.2 users on > > modern > > operating systems, especially given its LTS status (which OpenSSL hasn't > >

Re: Cutting support for OpenSSL 1.0.1 and 1.0.2 in 17~?

2023-09-07 Thread Thomas Munro
On Wed, May 24, 2023 at 11:03 PM Daniel Gustafsson wrote: > > On 24 May 2023, at 11:52, Michael Paquier wrote: > > On Wed, May 24, 2023 at 11:36:56AM +0200, Daniel Gustafsson wrote: > >> 1.0.2 is also an LTS version available commercially for premium support > >> customers of OpenSSL (1.1.1 will

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-09-06 Thread Thomas Munro
ssing consistently, which would be nice. From cbab5797fde1db70464f157b4bb9321ab40ad025 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 7 Sep 2023 15:28:47 +1200 Subject: [PATCH] Update recovery conflict stats immediately. 031_recovery_conflict.pl would occasionally fail to see a st

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-09-06 Thread Thomas Munro
On Sun, Aug 13, 2023 at 9:00 AM Andres Freund wrote: > On 2023-08-12 15:50:24 +1200, Thomas Munro wrote: > > Thanks. I realised that it's easy enough to test that theory about > > cleanup locks by hacking ConditionalLockBufferForCleanup() to return > > false randomly. Then

pgsql: Fix instability in 031_recovery_conflict.pl.

2023-09-06 Thread Thomas Munro
Fix instability in 031_recovery_conflict.pl. Where the test wants a VACUUM command to generate WAL that would conflict with a session on the standby, it could transiently fail to do so if it couldn't acquire a cleanup lock conditionally at that moment on the primary. VACUUM FREEZE will wait, so

pgsql: Fix recovery conflict SIGUSR1 handling.

2023-09-06 Thread Thomas Munro
Fix recovery conflict SIGUSR1 handling. We shouldn't be doing non-trivial work in signal handlers in general, and in this case the handler could reach unsafe code and corrupt state. It also clobbered its own "reason" code. Move all recovery conflict decision logic into the next

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-09-06 Thread Thomas Munro
I have now disabled the test in 15 and 16 (like the older branches). I'll see about getting the fixes into master today, and we can contemplate back-patching later, after we've collected a convincing volume of test results from the build farm, CI and hopefully your s390x master snapshot builds (if

pgsql: Disable 031_recovery_conflict.pl in 15 and 16.

2023-09-06 Thread Thomas Munro
Disable 031_recovery_conflict.pl in 15 and 16. This test fails due to known bugs in the test and the server. Those will be fixed in master shortly and possibly back-patched a bit later, but in the meantime it is unhelpful for package maintainers if the tests randomly fail, and it's not a good

<    1   2   3   4   5   6   7   8   9   10   >