Re: BF animal malleefowl reported an failure in 001_password.pl

2023-01-19 Thread Thomas Munro
On Tue, Jan 17, 2023 at 11:24 AM Thomas Munro wrote: > Another idea would be to teach the latch infrastructure itself to > magically swap latch events to position 0. Latches are usually > prioritised; it's only in this rare race case that they are not. I liked that idea for a whi

Re: BF animal malleefowl reported an failure in 001_password.pl

2023-01-16 Thread Thomas Munro
On Sun, Jan 15, 2023 at 12:35 AM Thomas Munro wrote: > Here's a sketch of the first idea. To hit the problem case, the signal needs to arrive in between the latch->is_set check and the epoll_wait() call, and the handler needs to take a while to get started. (If it arrives before the

Re: BF animal malleefowl reported an failure in 001_password.pl

2023-01-14 Thread Thomas Munro
On Sat, Jan 14, 2023 at 10:29 PM Thomas Munro wrote: > But if that's the general idea, I suppose there would be two ways to > give higher priority to signals/latches that arrive in the same set of > events: (1) scan the events array twice (for latches then > connections), or (2) check

Re: BF animal malleefowl reported an failure in 001_password.pl

2023-01-14 Thread Thomas Munro
On Sat, Jan 14, 2023 at 8:55 PM Tom Lane wrote: > "houzj.f...@fujitsu.com" writes: > > I noticed one BF failure[1] when monitoring the BF for some other commit. > > [1] > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=malleefowl=2023-01-13%2009%3A54%3A51 > > ... > > So it seems the

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2023-01-13 Thread Thomas Munro
ro-only too? But that's getting off track.) I split the patch in two: memory and interrupts. I also found a place in contrib/pg_trgm that did no-longer-needed try/finally. From 159d3d0fd7894c69f20367771b2100e48e064eec Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 4 Jan 2023 14:15:40 +1300 Su

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-13 Thread Thomas Munro
On Tue, Jan 10, 2023 at 3:05 PM Andres Freund wrote: > On 2023-01-03 12:05:20 -0800, Andres Freund wrote: > > The attached patch adds libpq-be-fe-helpers.h and uses it in > > libpqwalreceiver, > > dblink, postgres_fdw. > > > As I made libpq-be-fe-helpers.h handle reserving external fds, > >

Re: Missed condition-variable wakeups on FreeBSD

2023-01-12 Thread Thomas Munro
On Sun, Feb 27, 2022 at 8:07 AM Tom Lane wrote: > I have observed this three times in the REL_11 branch, once > in REL_12, and a couple of times last summer before it occurred > to me to start keeping notes. Over that time the machine has > been running various patchlevels of FreeBSD 13.0. FTR

PG_SETMASK() archeology

2023-01-12 Thread Thomas Munro
ght as well drop the macro to make our source a tiny bit less weird for new players, with a slightly richer interface. Trivial patch attached. From 1233157f7e7e0ea8c36bc4eaaf3ed2c89f3541df Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Mon, 9 Jan 2023 12:10:11 +1300 Subject: [PATCH] Retire PG_SETM

Experimenting with Postmaster variable scope

2023-01-12 Thread Thomas Munro
urn costs. From acc6b4a8961b93c7f07a09181a4369c01a2685a8 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 27 Dec 2022 03:46:56 +1300 Subject: [PATCH] Move postmaster's state out of global variables. Historically, the postmaster's state for tracking its children had to live in global variables, because it had to be accessi

Re: Using WaitEventSet in the postmaster

2023-01-12 Thread Thomas Munro
On Fri, Jan 13, 2023 at 7:26 AM Andres Freund wrote: > On 2023-01-12 20:35:43 +1300, Thomas Munro wrote: > > Subject: [PATCH] Fix WaitEventSetWait() buffer overrun. > Makes sense. We should backpatch this, I think? Done.

pgsql: Fix WaitEventSetWait() buffer overrun.

2023-01-12 Thread Thomas Munro
Fix WaitEventSetWait() buffer overrun. The WAIT_USE_EPOLL and WAIT_USE_KQUEUE implementations of WaitEventSetWaitBlock() confused the size of their internal buffer with the size of the caller's output buffer, and could ask the kernel for too many events. In fact the set of events retrieved from

pgsql: Fix WaitEventSetWait() buffer overrun.

2023-01-12 Thread Thomas Munro
Fix WaitEventSetWait() buffer overrun. The WAIT_USE_EPOLL and WAIT_USE_KQUEUE implementations of WaitEventSetWaitBlock() confused the size of their internal buffer with the size of the caller's output buffer, and could ask the kernel for too many events. In fact the set of events retrieved from

pgsql: Fix WaitEventSetWait() buffer overrun.

2023-01-12 Thread Thomas Munro
Fix WaitEventSetWait() buffer overrun. The WAIT_USE_EPOLL and WAIT_USE_KQUEUE implementations of WaitEventSetWaitBlock() confused the size of their internal buffer with the size of the caller's output buffer, and could ask the kernel for too many events. In fact the set of events retrieved from

pgsql: Fix WaitEventSetWait() buffer overrun.

2023-01-12 Thread Thomas Munro
Fix WaitEventSetWait() buffer overrun. The WAIT_USE_EPOLL and WAIT_USE_KQUEUE implementations of WaitEventSetWaitBlock() confused the size of their internal buffer with the size of the caller's output buffer, and could ask the kernel for too many events. In fact the set of events retrieved from

pgsql: Fix WaitEventSetWait() buffer overrun.

2023-01-12 Thread Thomas Munro
Fix WaitEventSetWait() buffer overrun. The WAIT_USE_EPOLL and WAIT_USE_KQUEUE implementations of WaitEventSetWaitBlock() confused the size of their internal buffer with the size of the caller's output buffer, and could ask the kernel for too many events. In fact the set of events retrieved from

pgsql: Fix WaitEventSetWait() buffer overrun.

2023-01-12 Thread Thomas Munro
Fix WaitEventSetWait() buffer overrun. The WAIT_USE_EPOLL and WAIT_USE_KQUEUE implementations of WaitEventSetWaitBlock() confused the size of their internal buffer with the size of the caller's output buffer, and could ask the kernel for too many events. In fact the set of events retrieved from

Re: Using WaitEventSet in the postmaster

2023-01-11 Thread Thomas Munro
On Thu, Jan 12, 2023 at 7:57 PM Thomas Munro wrote: > On Thu, Jan 12, 2023 at 7:27 PM Tom Lane wrote: > > skink seems to have found a problem: > > > > ==2011873== VALGRINDERROR-BEGIN > > ==2011873== Syscall param epoll_wait(events) points to unaddressable byte(s) >

Re: Using WaitEventSet in the postmaster

2023-01-11 Thread Thomas Munro
On Thu, Jan 12, 2023 at 7:27 PM Tom Lane wrote: > skink seems to have found a problem: > > ==2011873== VALGRINDERROR-BEGIN > ==2011873== Syscall param epoll_wait(events) points to unaddressable byte(s) > ==2011873==at 0x4D8DC73: epoll_wait (epoll_wait.c:30) > ==2011873==by 0x55CA49:

Re: Using WaitEventSet in the postmaster

2023-01-11 Thread Thomas Munro
On Wed, Jan 11, 2023 at 4:07 PM Thomas Munro wrote: > Thanks. Here's v10. I'll wait a bit longer to see if anyone else has > feedback. Pushed, after a few very minor adjustments, mostly comments. Thanks for the reviews and pointers. I think there are quite a lot of refac

pgsql: Use WaitEventSet API for postmaster's event loop.

2023-01-11 Thread Thomas Munro
Use WaitEventSet API for postmaster's event loop. Switch to a design similar to regular backends, instead of the previous arrangement where signal handlers did non-trivial state management and called fork(). The main changes are: * The postmaster now has its own local latch to wait on. (For

pgsql: Refactor DetermineSleepTime() to use milliseconds.

2023-01-11 Thread Thomas Munro
Refactor DetermineSleepTime() to use milliseconds. Since we're not using select() anymore, we don't need to bother with struct timeval. We can work directly in milliseconds, which the latch API wants. Discussion:

Re: [PATCH] psql: Add tab-complete for optional view parameters

2023-01-11 Thread Thomas Munro
On Thu, Jan 12, 2023 at 5:50 AM vignesh C wrote: > For some reason CFBot is not able to apply the patch, please have a > look and post an updated version if required, also check and handle > Dean Rasheed and Jim Jones comment if applicable: > === Applying patches on top of PostgreSQL commit ID >

Re: delay starting WAL receiver

2023-01-10 Thread Thomas Munro
On Wed, Jan 11, 2023 at 5:20 PM Thomas Munro wrote: > (I don't know why you didn't make it 0) (Oh, I see why it had to be non-zero to avoiding burning CPU, ignore that part.)

Re: delay starting WAL receiver

2023-01-10 Thread Thomas Munro
On Wed, Jan 11, 2023 at 2:08 PM Nathan Bossart wrote: > I discussed this a bit in a different thread [0], but I thought it deserved > its own thread. > > After setting wal_retrieve_retry_interval to 1ms in the tests, I noticed > that the recovery tests consistently take much longer. Upon further

Re: Using WaitEventSet in the postmaster

2023-01-10 Thread Thomas Munro
On Sun, Jan 8, 2023 at 11:55 AM Andres Freund wrote: > On 2023-01-07 18:08:11 +1300, Thomas Munro wrote: > > On Sat, Jan 7, 2023 at 12:25 PM Andres Freund wrote: > > > On 2023-01-07 11:08:36 +1300, Thomas Munro wrote: > > > > 3. Is it OK to clobber the shared pen

Re: Use windows VMs instead of windows containers on the CI

2023-01-10 Thread Thomas Munro
On Wed, Jan 11, 2023 at 8:20 AM Andres Freund wrote: > On 2023-01-10 09:22:12 -0600, Justin Pryzby wrote: > > > There is more than 2x speed gain when VMs are used. > > > > One consideration is that if windows runs twice as fast, we'll suddenly > > start using twice as many resources at

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-09 Thread Thomas Munro
On Tue, Jan 10, 2023 at 8:34 AM Andres Freund wrote: > A different approach would be to represent fxids as *signed* 64bit > integers. That'd of course loose more range, but could represent everything > accurately, and would have a compatible on-disk representation on two's > complement platforms

Re: Using WaitEventSet in the postmaster

2023-01-06 Thread Thomas Munro
On Sat, Jan 7, 2023 at 12:25 PM Andres Freund wrote: > On 2023-01-07 11:08:36 +1300, Thomas Munro wrote: > > 3. Is it OK to clobber the shared pending flag for SIGQUIT, SIGTERM, > > SIGINT? If you send all of these extremely rapidly, it's > > indeterminate w

Re: Using WaitEventSet in the postmaster

2023-01-06 Thread Thomas Munro
On Fri, Dec 23, 2022 at 8:46 PM Thomas Munro wrote: > I pushed the small preliminary patches. Here's a rebase of the main patch. Here are some questions I have considered. Anyone got an opinion on point 3, in particular? 1. Is it OK that we are now using APIs that might throw, in pla

Re: Cygwin cleanup

2023-01-06 Thread Thomas Munro
On Sat, Jan 7, 2023 at 3:40 AM Andrew Dunstan wrote: > OK, should I now try re-enabling TAP tests on lorikeet? Not before https://commitfest.postgresql.org/41/4032/ is committed. After that, it might be worth a try? I have no idea if the PANIC problem I mentioned last night would apply to

Re: Cygwin cleanup

2023-01-06 Thread Thomas Munro
On Wed, Nov 9, 2022 at 2:04 PM Justin Pryzby wrote: > +data_sync_retry = on Sharing with the list some clues that Justin and I figured out about what that part is doing. Without it, you get failures like: PANIC: could not open file "pg_logical/snapshots/0-14FE6B0.snap": No such file or

Re: pg_ftruncate hardcodes length=0 but only under windows

2023-01-05 Thread Thomas Munro
On Fri, Jan 6, 2023 at 4:16 PM Justin Pryzby wrote: > - ret = ftruncate(fd, 0); > + ret = ftruncate(fd, length); Oops. Right. Thanks, pushed.

pgsql: Fix pg_truncate() on Windows.

2023-01-05 Thread Thomas Munro
Fix pg_truncate() on Windows. Commit 57faaf376 added pg_truncate(const char *path, off_t length), but "length" was ignored under WIN32 and the file was unconditionally truncated to 0. There was no live bug, since the only caller passes 0. Fix, and back-patch to 14 where the function arrived.

pgsql: Fix pg_truncate() on Windows.

2023-01-05 Thread Thomas Munro
Fix pg_truncate() on Windows. Commit 57faaf376 added pg_truncate(const char *path, off_t length), but "length" was ignored under WIN32 and the file was unconditionally truncated to 0. There was no live bug, since the only caller passes 0. Fix, and back-patch to 14 where the function arrived.

pgsql: Fix pg_truncate() on Windows.

2023-01-05 Thread Thomas Munro
Fix pg_truncate() on Windows. Commit 57faaf376 added pg_truncate(const char *path, off_t length), but "length" was ignored under WIN32 and the file was unconditionally truncated to 0. There was no live bug, since the only caller passes 0. Fix, and back-patch to 14 where the function arrived.

Re: Cygwin cleanup

2023-01-05 Thread Thomas Munro
On Fri, Jan 6, 2023 at 1:22 AM Thomas Munro wrote: > https://cirrus-ci.com/task/5142810819559424 [still running at time of writing] > > Gotta run, but I'll check again in the morning to see if that does better... Yes! Two successful runs with all TAP tests so far. So it looks li

pgsql: Use unnamed POSIX semaphores on Cygwin.

2023-01-05 Thread Thomas Munro
Use unnamed POSIX semaphores on Cygwin. Testing on CI showed that Cygwin's semctl() can fail with EAGAIN (possibly due to resource limits in cygserver that could be tuned, not examined). Switch to so-called POSIX semaphores instead, which don't seem to fail in that way (possibly due to a more

Re: Cygwin cleanup

2023-01-05 Thread Thomas Munro
On Wed, Jan 4, 2023 at 3:25 AM Justin Pryzby wrote: > On Tue, Jan 03, 2023 at 05:54:56PM +0530, vignesh C wrote: > > Is there still some work pending for this thread as Andres had > > committed some part, if so, can you post an updated patch for the > > same. > > Thomas, what's your opinion ?

Re: pg_upgrade test failure

2023-01-04 Thread Thomas Munro
On Wed, Dec 7, 2022 at 7:15 AM Andres Freund wrote: > On 2022-11-08 01:16:09 +1300, Thomas Munro wrote: > > So [1] on its own didn't fix this. My next guess is that the attached > > might help. > What is our plan here? This afaict is the most common "false positive&q

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2023-01-04 Thread Thomas Munro
On Thu, Jan 5, 2023 at 11:55 AM Tom Lane wrote: > Andres Freund writes: > > Hm. Seems confusing for this to continue being called rcancelrequested() and > > to be called via if(CANCEL_REQUESTED()), if we're not even documenting that > > it's intended to be usable that way? > > Yeah. I'm not

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2023-01-04 Thread Thomas Munro
On Thu, Jan 5, 2023 at 12:33 PM Andres Freund wrote: > What about using a version of errsave() that can save FATALs too? We could > have something roughly like the ProcessInterrupts() in the proposed patch that > is used from within rcancelrequested(). But instead of actually throwing the >

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2023-01-03 Thread Thomas Munro
On Mon, Jan 2, 2023 at 8:38 AM Thomas Munro wrote: > On Sat, Dec 31, 2022 at 6:36 PM Noah Misch wrote: > > On Sat, Dec 31, 2022 at 10:06:53AM +1300, Thomas Munro wrote: > > > Idea #8 is a realisation that twisting oneself into a pretzel to avoid > > > having to ch

Re: RADIUS tests and improvements

2023-01-03 Thread Thomas Munro
On Wed, Jan 4, 2023 at 10:03 AM Andreas Karlsson wrote: > On 1/3/23 04:11, Thomas Munro wrote: > > [...] While adding > > the GUC I couldn't help wondering why RADIUS even needs a timeout > > separate from authentication_timeout; another way to go here would be > &

Re: RADIUS tests and improvements

2023-01-03 Thread Thomas Munro
On Wed, Jan 4, 2023 at 10:07 AM Andreas Karlsson wrote: > Another thing: shouldn't we set some wait event to indicate that we are > waiting the RADIUS server or is that pointless during authentication > since there are no queries running anyway? I initially added a wait_event value, but I

RADIUS tests and improvements

2023-01-02 Thread Thomas Munro
systems; if you know the answer there please LMK. From 72a63c87b595a31f3d5e68722ca6bc7460190cc0 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 31 Dec 2022 14:41:57 +1300 Subject: [PATCH 1/3] Add simple test for RADIUS authentication. This is similar to the existing tests for ldap and kerberos

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2023-01-01 Thread Thomas Munro
On Sat, Dec 31, 2022 at 6:36 PM Noah Misch wrote: > On Sat, Dec 31, 2022 at 10:06:53AM +1300, Thomas Munro wrote: > > Idea #8 is a realisation that twisting oneself into a pretzel to avoid > > having to change the regexp code or its REG_CANCEL control flow may be > > a bi

Re: Add a test to ldapbindpasswd

2023-01-01 Thread Thomas Munro
On Mon, Jan 2, 2023 at 3:04 AM Andrew Dunstan wrote: > On 2022-12-19 Mo 11:16, Andrew Dunstan wrote: > > There is currently no test for the use of ldapbindpasswd in the > > pg_hba.conf file. This patch, mostly the work of John Naylor, remedies that. > > > > > > This currently has failures on the

pgsql: ci: Change macOS builds from Intel to ARM.

2022-12-31 Thread Thomas Munro
ci: Change macOS builds from Intel to ARM. Cirrus is about to shut down its macOS-on-Intel support, so it's time to move our CI testing over to ARM instances. The Homebrew package manager changed its default installation prefix for the new architecture, so a couple of tests need tweaks to find

pgsql: ci: Change macOS builds from Intel to ARM.

2022-12-31 Thread Thomas Munro
ci: Change macOS builds from Intel to ARM. Cirrus is about to shut down its macOS-on-Intel support, so it's time to move our CI testing over to ARM instances. The Homebrew package manager changed its default installation prefix for the new architecture, so a couple of tests need tweaks to find

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2022-12-30 Thread Thomas Munro
On Thu, Dec 29, 2022 at 9:40 PM Noah Misch wrote: > On Tue, Jul 26, 2022 at 07:22:52PM -0400, Tom Lane wrote: > > Thomas Munro writes: > > > ... The regular expression machinery is capable of > > > consuming a lot of CPU, and does CANCEL_REQUESTED(nfa->v->re)

Re: CI and test improvements

2022-12-29 Thread Thomas Munro
/message-id/flat/CAN55FZ2R%2BXufuVgJ8ew_yDBk48PgXEBvyKNvnNdTTVyczbQj0g%40mail.gmail.com From a355638fd9c1c16e54418d8374a374967081e0a0 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 28 Dec 2022 16:58:09 +1300 Subject: [PATCH] ci: Change macOS builds from Intel to ARM. Cirrus is about to shu

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2022-12-29 Thread Thomas Munro
On Fri, Dec 30, 2022 at 10:54 AM Andres Freund wrote: > On 2022-12-30 10:31:22 +1300, Thomas Munro wrote: > > On Fri, Dec 30, 2022 at 6:23 AM Andres Freund wrote: > > > On 2022-12-08 18:08:15 -0800, Andres Freund wrote: > > > > On 2022-09-25 16:22:37 -0700, Andres

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2022-12-29 Thread Thomas Munro
On Fri, Dec 30, 2022 at 6:23 AM Andres Freund wrote: > On 2022-12-08 18:08:15 -0800, Andres Freund wrote: > > On 2022-09-25 16:22:37 -0700, Andres Freund wrote: > > The only alternative way to provide a wrapper that I can think of are to > > a) introduce a new static library that can be linked to

Re: Using WaitEventSet in the postmaster

2022-12-22 Thread Thomas Munro
I pushed the small preliminary patches. Here's a rebase of the main patch. From d23fba75cf693ffabc068a36424b7be22342c1b2 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 9 Nov 2022 22:59:58 +1300 Subject: [PATCH v7] Give the postmaster a WaitEventSet and a latch. Traditionally

pgsql: Don't leak a signalfd when using latches in the postmaster.

2022-12-22 Thread Thomas Munro
Don't leak a signalfd when using latches in the postmaster. At the time of commit 6a2a70a02 we didn't use latch infrastructure in the postmaster. We're planning to start doing that, so we'd better make sure that the signalfd inherited from a postmaster is not duplicated and then leaked in the

pgsql: Allow parent's WaitEventSets to be freed after fork().

2022-12-22 Thread Thomas Munro
Allow parent's WaitEventSets to be freed after fork(). An epoll fd belonging to the parent should be closed in the child. A kqueue fd is automatically closed by fork(), but we should still adjust our counter. For poll and Windows systems, nothing special is required. On all systems we free the

pgsql: Add WL_SOCKET_ACCEPT event to WaitEventSet API.

2022-12-22 Thread Thomas Munro
Add WL_SOCKET_ACCEPT event to WaitEventSet API. To be able to handle incoming connections on a server socket with the WaitEventSet API, we'll need a new kind of event to indicate that the the socket is ready to accept a connection. On Unix, it's just the same as WL_SOCKET_READABLE, but on

Re: Windows default locale vs initdb

2022-12-22 Thread Thomas Munro
On Fri, Jul 29, 2022 at 3:33 PM Thomas Munro wrote: > On Fri, Jul 22, 2022 at 11:59 PM Juan José Santamaría Flecha > wrote: > > TL;DR; What I want to show through this example is that Windows ACP is not > > modified by setlocale(), it can only be done through the Windows regi

Re: Array initialisation notation in syscache.c

2022-12-22 Thread Thomas Munro
On Thu, Dec 22, 2022 at 5:36 AM Peter Eisentraut wrote: > This looks like a good improvement to me. Thanks both. Pushed. > (I have also thought about having this generated from the catalog > definition files somehow, but one step at a time ...) Good plan.

pgsql: Improve notation of cacheinfo table in syscache.c.

2022-12-22 Thread Thomas Munro
Improve notation of cacheinfo table in syscache.c. Use C99 designated initializer syntax for the array elements, instead of writing the enumerator name and position in a comment. Replace nkeys and key with a local variadic macro, for a shorter notation. Reviewed-by: Tom Lane Reviewed-by: Peter

Re: psql tab-complete

2022-12-21 Thread Thomas Munro
On Thu, Dec 22, 2022 at 3:25 PM Michael Paquier wrote: > On Wed, Dec 21, 2022 at 05:56:05PM -0500, Tom Lane wrote: > > Thomas Munro writes: > >> Since I really want to be able to use VA_ARGS_NARGS() elsewhere, > > > > +1, that'd be handy. > > > >&

pgsql: Add work-around for VA_ARGS_NARGS() on MSVC.

2022-12-21 Thread Thomas Munro
remember to clean this up in the future when our minimum MSVC version advances. Author: Victor Spirin Reviewed-by: Thomas Munro Reviewed-by: Tom Lane Reviewed-by: Michael Paquier Discussion: https://postgr.es/m/f450fc57-a147-19d0-e50c-33571c52cc13%40postgrespro.ru Branch -- master Details

Re: Direct I/O

2022-12-21 Thread Thomas Munro
On Wed, Dec 14, 2022 at 5:48 PM Thomas Munro wrote: > 0001 -- David's palloc_aligned() patch > https://commitfest.postgresql.org/41/3999/ > 0002 -- I/O-align almost all buffers used for I/O > 0003 -- Add the GUCs > 0004 -- Throwaway hack to make cfbot turn the GUCs on David p

Re: psql tab-complete

2022-12-21 Thread Thomas Munro
On Sat, Oct 26, 2019 at 4:59 PM Michael Paquier wrote: > On Fri, Oct 25, 2019 at 11:57:18AM +0300, Victor Spirin wrote: > > This patch resolved one problem in the tab-complete.c on MSVC. The > > VA_ARGS_NARGS macros now work correctly on Windows. > > Can you explain why and in what the use of

Re: Array initialisation notation in syscache.c

2022-12-20 Thread Thomas Munro
On Wed, Dec 21, 2022 at 1:33 PM Thomas Munro wrote: > KEY(Anum_pg_attribute_attrelid, > Anum_pg_attribute_attnum), I independently rediscovered that our VA_ARGS_NARGS() macro in c.h always returns 1 on MSVC via trial-by-CI. Derp. Here is the same patch, no change f

Re: Array initialisation notation in syscache.c

2022-12-20 Thread Thomas Munro
On Wed, Dec 21, 2022 at 12:05 PM Tom Lane wrote: > Thomas Munro writes: > > Do you think this is better? > > I'm not at all on board with adding runtime overhead to > save maintaining the nkeys fields. I don't see how to do it at compile time without getting the preprocessor

Re: meson files copyright

2022-12-20 Thread Thomas Munro
On Tue, Dec 20, 2022 at 6:27 PM Noah Misch wrote: > On Mon, Dec 19, 2022 at 09:09:25PM +0100, Peter Eisentraut wrote: > > On 19.12.22 19:33, Robert Haas wrote: > > >On Mon, Dec 19, 2022 at 1:03 PM Tom Lane wrote: > > >>Vik Fearing writes: > > >>>Perhaps a bit off-topic, but what is the point of

Array initialisation notation in syscache.c

2022-12-20 Thread Thomas Munro
, but it's not a complicated struct that the eye gets lost in, so I didn't bother with that in the attached. From 3c554145b65bde834a1ff9f65396d2ab07f72120 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 20 Dec 2022 15:47:59 +1300 Subject: [PATCH] Improve notation of cacheinfo table in syscache.c

Re: Fwd: temp_file_limit?

2022-12-19 Thread Thomas Munro
On Tue, Dec 20, 2022 at 8:59 AM Frits Jalvingh wrote: > @ranier > These files ONLY exist during the query. They get deleted as soon as the > query terminates, by Postgres itself. Once the query terminates pgsql_tmp is > completely empty. Considering what Thomas said (and the actual occurrence

Re: Fwd: temp_file_limit?

2022-12-18 Thread Thomas Munro
On Mon, Dec 19, 2022 at 1:51 PM Thomas Munro wrote: > It's really the limit for a single file Oops, sorry I take that back. It should be per process.

Re: Fwd: temp_file_limit?

2022-12-18 Thread Thomas Munro
On Mon, Dec 19, 2022 at 9:11 AM Justin Pryzby wrote: > On Sun, Dec 18, 2022 at 06:29:41PM +0100, Frits Jalvingh wrote: > > Just to be clear: my real question is: why is temp_file_limit not > > working at the specified size? Because this is my real problem: when a > > query is dying like this it

Re: check_strxfrm_bug()

2022-12-17 Thread Thomas Munro
On Thu, Dec 15, 2022 at 3:22 PM Thomas Munro wrote: > ... If you're worried that the bugs might > come back, then the test is insufficient: modern versions of both OSes > have strxfrm_l(), which we aren't checking. With my garbage collector hat on, that made me wonder if there was

Re: pgsql: meson: Add 'running' test setup, as a replacement for installche

2022-12-15 Thread Thomas Munro
On Thu, Dec 8, 2022 at 9:48 AM Andres Freund wrote: > meson: Add 'running' test setup, as a replacement for installcheck The -running tests don't seem to be completely stable on CI. I don't know why (smells like plan changes?) but see cfbot.cputube.org for some flapping on FreeBSD (the only OS

pgsql: Fix typo in reference to __FreeBSD__.

2022-12-15 Thread Thomas Munro
Fix typo in reference to __FreeBSD__. Commit a2a8acd152 introduced a platform-dependent mechanism to prevent developers from referencing errno in the argument list of elog()/ereport(), but didn't use the right macro to detect FreeBSD, so it didn't actually work there. Reported-by: Japin Li

Re: Typo macro name on FreeBSD?

2022-12-15 Thread Thomas Munro
On Fri, Dec 16, 2022 at 4:44 PM Japin Li wrote: > Recently, I compile PostgreSQL on FreeBSD, I find commit a2a8acd152 introduecs > __freebsd__ macro, however, I cannot find this macro on FreeBSD 13. There only > has __FreeBSD__ macro. Is this a typo? Yeah, that seems to be my fault. Will fix.

check_strxfrm_bug()

2022-12-14 Thread Thomas Munro
would guess that a couple of other OSes with the old Berkeley locale code are similar. From 8a70596a7de680e02fb2a6db59e622a653f097c2 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 15 Dec 2022 13:38:57 +1300 Subject: [PATCH] Remove obsolete defense against strxfrm() bugs. Old versions

Re: [PATCH] Enable using llvm jitlink as an alternative llvm jit linker of old Rtdyld.

2022-12-14 Thread Thomas Munro
On Thu, Nov 24, 2022 at 12:08 AM David Rowley wrote: > On Wed, 23 Nov 2022 at 23:13, Alex Fan wrote: > > I am new to the postgres community and apologise for resending this as the > > previous one didn't include patch properly and didn't cc reviewers (maybe > > the reason it has been buried in

Re: Direct I/O

2022-12-13 Thread Thomas Munro
On Wed, Nov 2, 2022 at 11:54 AM Andres Freund wrote: > On 2022-11-02 09:44:30 +1300, Thomas Munro wrote: > > On Wed, Nov 2, 2022 at 2:33 AM Justin Pryzby wrote: > > > On Tue, Nov 01, 2022 at 08:36:18PM +1300, Thomas Munro wrote: > > > > io_data_direct = whether

Re: Tree-walker callbacks vs -Wdeprecated-non-prototype

2022-12-12 Thread Thomas Munro
On Mon, Dec 12, 2022 at 4:43 PM Thomas Munro wrote: > On Mon, Dec 12, 2022 at 4:07 PM Tom Lane wrote: > > I'm for "turn the warning off". Per previous discussion, adhering > > strictly to that rule would make our code worse (less legible AND > > less safe), not

pgsql: Disable clang 16's -Wcast-function-type-strict.

2022-12-12 Thread Thomas Munro
Disable clang 16's -Wcast-function-type-strict. Clang 16 is still in development, but seawasp reveals that it has started warning about many of our casts of function pointers (those introduced by commit 1c27d16e, and some older ones). Disable the new warning for now, since otherwise buildfarm

Re: Tree-walker callbacks vs -Wdeprecated-non-prototype

2022-12-11 Thread Thomas Munro
On Mon, Dec 12, 2022 at 4:07 PM Tom Lane wrote: > Thomas Munro writes: > > As visible on seawasp and locally (16/main branch nightly packages), > > they decided to start warning about these casts with a new strict > > variant of the warning. Their discussion: > >

Re: Tree-walker callbacks vs -Wdeprecated-non-prototype

2022-12-11 Thread Thomas Munro
As visible on seawasp and locally (16/main branch nightly packages), they decided to start warning about these casts with a new strict variant of the warning. Their discussion: https://reviews.llvm.org/D134831 There are also a few other cases unrelated to this thread's original problem, for

Re: -Wunreachable-code-generic-assoc warnings on elver

2022-12-10 Thread Thomas Munro
On Sat, Dec 10, 2022 at 3:50 PM Tom Lane wrote: > Recently, buildfarm member elver has started spewing literally > thousands of $SUBJECT: > > elver | 2022-12-10 01:17:29 | > ../../src/include/utils/float.h:223:33: warning: due to lvalue conversion of > the controlling expression,

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
Oops, v5 was broken as visible on cfbot (a last second typo broke it). Here's a better one. From 07b04dc410118ad04fd0006edda7ba80f241357a Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 6 Dec 2022 15:21:11 +1300 Subject: [PATCH v6 1/4] Add WL_SOCKET_ACCEPT event to WaitEventSet API

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
On Wed, Dec 7, 2022 at 2:08 PM Justin Pryzby wrote: > > + /* > > + * It would probably be safe to re-use the inherited signalfd > > since > > + * signalfds only see the current processes pending signals, > > but it > > I think you mean "current process's",

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
On Wed, Dec 7, 2022 at 12:12 PM Andres Freund wrote: > On 2022-12-07 00:58:06 +1300, Thomas Munro wrote: > > One way to fix that for the epoll version is to EPOLL_CTL_DEL and > > EPOLL_CTL_ADD, whenever transitioning to/from a zero event mask. > > Tried like that in t

Re: Cygwin cleanup

2022-12-06 Thread Thomas Munro
On Fri, Jul 29, 2022 at 10:57 AM Thomas Munro wrote: > I wonder if these problems would go away as a nice incidental > side-effect if we used latches for postmaster wakeups. I don't > know... maybe, if the problem is just with the postmaster's pattern of > blocking/unblocking? M

Re: Using WaitEventSet in the postmaster

2022-12-06 Thread Thomas Munro
On Tue, Dec 6, 2022 at 7:09 AM Andres Freund wrote: > On 2022-12-05 22:45:57 +1300, Thomas Munro wrote: > > The reason for the existing sleep-based approach was that we didn't > > want to accept any more connections (or spin furiously because the > > listen queu

Re: Collation version tracking for macOS

2022-12-05 Thread Thomas Munro
On Tue, Dec 6, 2022 at 6:45 AM Joe Conway wrote: > On 12/5/22 12:41, Jeff Davis wrote: > > On Mon, 2022-12-05 at 16:12 +1300, Thomas Munro wrote: > >> 1. I think we should seriously consider provider = ICU63. I still > >> think search-by-collversion is a litt

Re: Using WaitEventSet in the postmaster

2022-12-05 Thread Thomas Munro
On Sat, Dec 3, 2022 at 10:41 AM Thomas Munro wrote: > Here's an iteration like that. Still WIP grade. It passes, but there > must be something I don't understand about this computer program yet, > because if I move the "if (pending_..." section up into the block > where WL

Re: Collation version tracking for macOS

2022-12-04 Thread Thomas Munro
On Tue, Nov 29, 2022 at 7:51 PM Jeff Davis wrote: > On Sat, 2022-11-26 at 18:27 +1300, Thomas Munro wrote: > > On Thu, Nov 24, 2022 at 5:48 PM Thomas Munro > > wrote: > > > On Thu, Nov 24, 2022 at 3:07 PM Jeff Davis > > > wrote: > > > > I'd vote for

Re: Using WaitEventSet in the postmaster

2022-12-02 Thread Thomas Munro
On Fri, Dec 2, 2022 at 3:36 PM Thomas Munro wrote: > On Fri, Dec 2, 2022 at 2:40 PM Andres Freund wrote: > > It doesn't seem trivial (but not impossible either) to make SetLatch() > > robust > > against arbitrary corruption. So it seems easier to me to just put the latch

Re: Using WaitEventSet in the postmaster

2022-12-01 Thread Thomas Munro
On Fri, Dec 2, 2022 at 2:40 PM Andres Freund wrote: > On 2022-12-02 10:12:25 +1300, Thomas Munro wrote: > > with a latch as the wakeup mechanism for "PM signals" (requests from > > backends to do things like start a background worker, etc). > > Hm - is that directly

Re: pgsql: Expand AclMode to 64 bits

2022-12-01 Thread Thomas Munro
On Sat, Nov 26, 2022 at 3:00 AM Andrew Dunstan wrote: > >>> crake is complaining for the upgrades from v12: > >>> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake=2022-11-23%2023%3A32%3A04 > >>> > >>> It seems that there are some tables dependent on aclitem, bumping on > >>> your

Using AF_UNIX sockets always for tests on Windows

2022-12-01 Thread Thomas Munro
9w5Fgzdr4G8AnEtJ%3Dz%3Dp6hGDEm4qYGEUX5B6fQ%40mail.gmail.com From 62b1cdbdc848f96eef02ed97f14be9c1f4557539 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 7 Sep 2022 07:35:11 +1200 Subject: [PATCH 1/4] WIP: Make mkdtemp() more secure on Windows. Our POSIX mkdtemp() implementation in src/port/mkdtemp

Using WaitEventSet in the postmaster

2022-12-01 Thread Thomas Munro
that line of thinking up for now.) From 978aa358885312372f842cd47549bb04a78477ab Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 9 Nov 2022 22:59:58 +1300 Subject: [PATCH] Give the postmaster a WaitEventSet and a latch. Traditionally, the postmaster's architecture was quite unusual. It di

Re: wake up logical workers after ALTER SUBSCRIPTION

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 5:23 PM Thomas Munro wrote: > On Wed, Nov 30, 2022 at 5:10 PM Nathan Bossart > wrote: > > I spent some more time on the prevent-unnecessary-wakeups patch for > > logical/worker.c that I've been alluding to in this thread, and I found a > > few mor

Re: wake up logical workers after ALTER SUBSCRIPTION

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 5:10 PM Nathan Bossart wrote: > I spent some more time on the prevent-unnecessary-wakeups patch for > logical/worker.c that I've been alluding to in this thread, and I found a > few more places where we depend on the worker periodically waking up. This > seems to be a

Re: Reducing power consumption on idle servers

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 1:32 AM Simon Riggs wrote: > Re-attaching patch for bgwriter and walwriter, so it is clear this is > not yet committed. I'm just curious, and not suggesting that 60s wakeups are a problem for the polar ice caps, but why even time out at all? Are the latch protocols

Re: Strange failure on mamba

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 3:43 PM Thomas Munro wrote: > sigaction(0, NULL, ) s/sigaction/sigprocmask/

<    6   7   8   9   10   11   12   13   14   15   >