Re: Commitfest 2023-03 starting tomorrow!

2023-03-21 Thread Thomas Munro
address --++-- 8d578b9b2e37a4d9d6f422ced5126acec62365a7 | 2023-03-21 14:29:34+13 | Melanie Plageman 8d578b9b2e37a4d9d6f422ced5126acec62365a7 | 2023-03-21 14:29:34+13 | Thomas Munro (2 rows) You can really only go back about 5-7 years befor

Re: Using AF_UNIX sockets always for tests on Windows

2023-03-21 Thread Thomas Munro
on feedback already given, not included in this revision. From e7fac7a15ed0eda6516e7fa0917c06e005341b00 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 7 Sep 2022 07:35:11 +1200 Subject: [PATCH v3 1/2] Make mkdtemp() more secure on Windows. MIME-Version: 1.0 Content-Type: text/plain; chars

Re: Assertion failure with barriers in parallel hash join

2023-03-20 Thread Thomas Munro
Pushed and back-patched, with minor comment tweaks. Apologies for taking so long.

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

Re: Commitfest 2023-03 starting tomorrow!

2023-03-20 Thread Thomas Munro
On Tue, Mar 21, 2023 at 9:22 AM Greg Stark wrote: > Yeah, even aside from flappers there's the problem that it's about as > common for real commits to break some test as it is for patches to > start failing tests. So often it's a real failure but it's nothing to > do with the patch, it has to do

Re: Commitfest 2023-03 starting tomorrow!

2023-03-20 Thread Thomas Munro
On Tue, Mar 21, 2023 at 3:15 AM Greg Stark wrote: > The next level of this would be something like notifying the committer > with a list of patches in the CF that a commit broke. I don't > immediately see how to integrate that with our workflow but I have > seen something like this work well in a

Re: Commitfest 2023-03 starting tomorrow!

2023-03-19 Thread Thomas Munro
On Mon, Mar 20, 2023 at 1:10 PM Tom Lane wrote: > Thomas Munro writes: > > ... I suppose then someone might complain > > that it should be clearer if a patch hasn't applied for a very long > > time; suggestions for how to show that are welcome. > > Can you make t

Re: Commitfest 2023-03 starting tomorrow!

2023-03-19 Thread Thomas Munro
On Mon, Mar 20, 2023 at 11:13 AM Thomas Munro wrote: > I realised that part of Alvaro's complaint was probably caused by > cfbot's refusal to show any useful information just because it > couldn't apply a patch the last time it tried. A small improvement > today: now it show

Re: Commitfest 2023-03 starting tomorrow!

2023-03-19 Thread Thomas Munro
On Sun, Mar 19, 2023 at 12:44 PM Justin Pryzby wrote: > On Sat, Mar 18, 2023 at 04:28:02PM -0700, Peter Geoghegan wrote: > > On Sat, Mar 18, 2023 at 4:19 PM Justin Pryzby wrote: > > > The only issue with this is that cfbot has squished all the commits into > > > one, and lost the original commit

Re: File descriptors in exec'd subprocesses

2023-03-17 Thread Thomas Munro
I pushed the libpq changes. I'll leave the pipe2 and accept4 changes on ice for now, maybe for a later cycle (unlike the committed patches, they don't currently fix a known problem, they just avoid some syscalls that are already fairly rare). For the libpq change, the build farm seems happy so

Re: src/test/examples/testlibpq2.c where the HAVE_SYS_SELECT_H is defined.

2023-03-17 Thread Thomas Munro
On Fri, Mar 17, 2023 at 7:48 PM jian he wrote: > Hi, > playing around with $[0] testlibpq2.c example. I wondered where > HAVE_SYS_SELECT_H is defined? > > I searched on the internet, founded that people also asked the same question > in $[1]. > > In my machine, I do have . > system version:

pgsql: libpq: Use modern socket flags, if available.

2023-03-17 Thread Thomas Munro
libpq: Use modern socket flags, if available. Since commit 7627b91cd5d, libpq has used FD_CLOEXEC so that sockets wouldn't be leaked to subprograms. With enough bad luck, a multi-threaded program might fork in between the socket() and fcntl() calls. We can close that tiny gap by using

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

Re: pgsql: Small tidyup for commit d41a178b.

2023-03-16 Thread Thomas Munro
On Fri, Mar 17, 2023 at 1:36 PM Tom Lane wrote: > Thomas Munro writes: > > Small tidyup for commit d41a178b. > > A comment was left behind claiming that we needed to use malloc() rather > > than palloc() because the corresponding free would run in another > > thread,

Re: slapd logs to syslog during tests

2023-03-16 Thread Thomas Munro
On Fri, Mar 17, 2023 at 9:15 AM Andrew Dunstan wrote: > On 2023-03-11 Sa 18:37, Andres Freund wrote: > On my buildfarm host (for all my animals) I noted that slapd was by far the > biggest contributor to syslog. Even though there's not normally slapd > running. It's of course the slapds started

pgsql: Small tidyup for commit d41a178b.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b. A comment was left behind claiming that we needed to use malloc() rather than palloc() because the corresponding free would run in another thread, but that's not true anymore. Remove that comment. And, with the reason being gone, we might as well actually use

pgsql: Small tidyup for commit d41a178b.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b. A comment was left behind claiming that we needed to use malloc() rather than palloc() because the corresponding free would run in another thread, but that's not true anymore. Remove that comment. And, with the reason being gone, we might as well actually use

pgsql: Small tidyup for commit d41a178b.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b. A comment was left behind claiming that we needed to use malloc() rather than palloc() because the corresponding free would run in another thread, but that's not true anymore. Remove that comment. And, with the reason being gone, we might as well actually use

pgsql: Small tidyup for commit d41a178b.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b. A comment was left behind claiming that we needed to use malloc() rather than palloc() because the corresponding free would run in another thread, but that's not true anymore. Remove that comment. And, with the reason being gone, we might as well actually use

pgsql: Small tidyup for commit d41a178b.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b. A comment was left behind claiming that we needed to use malloc() rather than palloc() because the corresponding free would run in another thread, but that's not true anymore. Remove that comment. And, with the reason being gone, we might as well actually use

pgsql: Small tidyup for commit d41a178b.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b. A comment was left behind claiming that we needed to use malloc() rather than palloc() because the corresponding free would run in another thread, but that's not true anymore. Remove that comment. And, with the reason being gone, we might as well actually use

Re: windows CI failing PMSignalState->PMChildFlags[slot] == PM_CHILD_ASSIGNED

2023-03-15 Thread Thomas Munro
On Thu, Mar 16, 2023 at 2:00 AM Alexander Lakhin wrote: > 15.03.2023 11:43, Thomas Munro wrote: > > Do you know how it fails in non-assert builds, without the fix? > So at least with my test script that doesn't lead to a crash or something. Thanks. We were wondering if the ret

Re: windows CI failing PMSignalState->PMChildFlags[slot] == PM_CHILD_ASSIGNED

2023-03-15 Thread Thomas Munro
On Wed, Mar 15, 2023 at 9:00 PM Alexander Lakhin wrote: > The result depends on some OS conditions (it reproduced pretty well > immediately after VM reboot), but it's enough to test the patch proposed. > And I can confirm that the Assert is not observed anymore (with the sleep > added after

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-14 Thread Thomas Munro
On Wed, Mar 15, 2023 at 7:54 AM Nathan Bossart wrote: > Here is roughly what I had in mind: > > NOTE: Although the delay is specified in microseconds, older Unixen > and > Windows use periodic kernel ticks to wake up, which might increase the > delay time significantly.

pgsql: Update obsolete comment about pg_usleep() accuracy.

2023-03-14 Thread Thomas Munro
Update obsolete comment about pg_usleep() accuracy. There are still some systems that use traditional tick-based sleep timing, but many including Linux, FreeBSD and macOS started using high resolution timer hardware more directly a decade or two ago. Update our comment about that. Also

pgsql: Use nanosleep() to implement pg_usleep().

2023-03-14 Thread Thomas Munro
Use nanosleep() to implement pg_usleep(). The previous coding based on select() had commentary about historical portability concerns. Use POSIX nanosleep() instead. This has independently been suggested a couple of times before, but never managed to stick. Since recent and proposed work

pgsql: Fix fractional vacuum_cost_delay.

2023-03-14 Thread Thomas Munro
Fix fractional vacuum_cost_delay. Commit 4753ef37 changed vacuum_delay_point() to use the WaitLatch() API, to fix the problem that vacuum could keep running for a very long time after the postmaster died. Unfortunately, that broke commit caf626b2's support for fractional vacuum_cost_delay, which

pgsql: Fix fractional vacuum_cost_delay.

2023-03-14 Thread Thomas Munro
Fix fractional vacuum_cost_delay. Commit 4753ef37 changed vacuum_delay_point() to use the WaitLatch() API, to fix the problem that vacuum could keep running for a very long time after the postmaster died. Unfortunately, that broke commit caf626b2's support for fractional vacuum_cost_delay, which

pgsql: Fix fractional vacuum_cost_delay.

2023-03-14 Thread Thomas Munro
Fix fractional vacuum_cost_delay. Commit 4753ef37 changed vacuum_delay_point() to use the WaitLatch() API, to fix the problem that vacuum could keep running for a very long time after the postmaster died. Unfortunately, that broke commit caf626b2's support for fractional vacuum_cost_delay, which

pgsql: Fix waitpid() emulation on Windows.

2023-03-14 Thread Thomas Munro
Fix waitpid() emulation on Windows. Our waitpid() emulation didn't prevent a PID from being recycled by the OS before the call to waitpid(). The postmaster could finish up tracking more than one child process with the same PID, and confuse them. Fix, by moving the guts of

pgsql: Fix waitpid() emulation on Windows.

2023-03-14 Thread Thomas Munro
Fix waitpid() emulation on Windows. Our waitpid() emulation didn't prevent a PID from being recycled by the OS before the call to waitpid(). The postmaster could finish up tracking more than one child process with the same PID, and confuse them. Fix, by moving the guts of

pgsql: Fix waitpid() emulation on Windows.

2023-03-14 Thread Thomas Munro
Fix waitpid() emulation on Windows. Our waitpid() emulation didn't prevent a PID from being recycled by the OS before the call to waitpid(). The postmaster could finish up tracking more than one child process with the same PID, and confuse them. Fix, by moving the guts of

pgsql: Fix waitpid() emulation on Windows.

2023-03-14 Thread Thomas Munro
Fix waitpid() emulation on Windows. Our waitpid() emulation didn't prevent a PID from being recycled by the OS before the call to waitpid(). The postmaster could finish up tracking more than one child process with the same PID, and confuse them. Fix, by moving the guts of

pgsql: Fix waitpid() emulation on Windows.

2023-03-14 Thread Thomas Munro
Fix waitpid() emulation on Windows. Our waitpid() emulation didn't prevent a PID from being recycled by the OS before the call to waitpid(). The postmaster could finish up tracking more than one child process with the same PID, and confuse them. Fix, by moving the guts of

pgsql: Fix waitpid() emulation on Windows.

2023-03-14 Thread Thomas Munro
Fix waitpid() emulation on Windows. Our waitpid() emulation didn't prevent a PID from being recycled by the OS before the call to waitpid(). The postmaster could finish up tracking more than one child process with the same PID, and confuse them. Fix, by moving the guts of

Re: windows CI failing PMSignalState->PMChildFlags[slot] == PM_CHILD_ASSIGNED

2023-03-14 Thread Thomas Munro
Here's a better version with more explicit comments about some details. It passes on CI. Planning to push this soon. From 0ac63e73313fffa19db22f19452c4bb08ea7bf14 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 14 Mar 2023 11:53:26 +1300 Subject: [PATCH v2] Fix waitpid() emulation

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-13 Thread Thomas Munro
On Tue, Mar 14, 2023 at 12:10 PM Nathan Bossart wrote: > > * NOTE: although the delay is specified in microseconds, the effective > > - * resolution is only 1/HZ, or 10 milliseconds, on most Unixen. Expect > > - * the requested delay to be rounded up to the next resolution boundary. > > + *

Re: windows CI failing PMSignalState->PMChildFlags[slot] == PM_CHILD_ASSIGNED

2023-03-13 Thread Thomas Munro
0/?p=11803 From cd83520c7d9ca214c98b9c0d4c74c9371fb0 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 14 Mar 2023 11:53:26 +1300 Subject: [PATCH] Fix waitpid() emulation on Windows. Unix doesn't allow a PID to be recycled for a new process until after the earlier process is "reaped" by waitpid() (o

Microsecond-based timeouts

2023-03-12 Thread Thomas Munro
on Windows with separate timer events, but I don't know. [1] https://www.postgresql.org/message-id/flat/CAAKRu_b-q0hXCBUCAATh0Z4Zi6UkiC0k2DFgoD3nC-r3SkR3tg%40mail.gmail.com From e99b7d31831f31888a9433a83d3e64ccbe2cc5c7 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 10 Mar 2023 15:16:47 +1300 Subject:

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-12 Thread Thomas Munro
On Sat, Mar 11, 2023 at 11:49 AM Tom Lane wrote: > Thomas Munro writes: > > I think this is the minimal back-patchable change. I propose to go > > ahead and do that, and then to kick the ideas about latch API changes > > into a new thread for the next commitfest. > &g

Re: pgsql: Add standard collation UNICODE

2023-03-12 Thread Thomas Munro
On Mon, Mar 13, 2023 at 9:10 AM Tom Lane wrote: > Andrew Dunstan writes: > > On 2023-03-11 Sa 15:47, Tom Lane wrote: > >> In general, I see no good reason for our regression tests to be making > >> assumptions about exactly how ICU's root locale behaves, so I'd suggest > >> just lobotomizing

Re: windows CI failing PMSignalState->PMChildFlags[slot] == PM_CHILD_ASSIGNED

2023-03-11 Thread Thomas Munro
And again: TRAP: failed Assert("PMSignalState->PMChildFlags[slot] == PM_CHILD_ASSIGNED"), File: "../src/backend/storage/ipc/pmsigna... https://cirrus-ci.com/task/6558324615806976

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-11 Thread Thomas Munro
On Fri, Mar 10, 2023 at 1:05 PM Thomas Munro wrote: > On Fri, Mar 10, 2023 at 11:37 AM Nathan Bossart > wrote: > > On Thu, Mar 09, 2023 at 05:27:08PM -0500, Tom Lane wrote: > > > Is it reasonable to assume that all modern platforms can time > > > millisecond delays

Re: Dead code in ps_status.c

2023-03-10 Thread Thomas Munro
On Fri, Feb 17, 2023 at 3:38 AM Tom Lane wrote: > Thomas Munro writes: > > On Thu, Feb 16, 2023 at 6:34 PM Tom Lane wrote: > > My GCC compile farm account seems to have expired, or something, so I > > couldn't check on wrasse's host (though whether wrasse is "live&

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-10 Thread Thomas Munro
On Fri, Mar 10, 2023 at 6:58 PM Thomas Munro wrote: > ... Perhaps something like 0004, which also shows the sort > of thing that we might consider back-patching to 14 and 15 (next > revision I'll move that up the front and put it in back-patchable > form). I think this is the

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-09 Thread Thomas Munro
On Fri, Mar 10, 2023 at 2:45 PM Thomas Munro wrote: > I'll go and see about usec latch waits. More soon. Here are some experimental patches along those lines. Seems good locally, but I saw a random failure I don't understand on CI so apparently I need to find a bug; at least this gives an i

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-09 Thread Thomas Munro
On Fri, Mar 10, 2023 at 2:21 PM Tom Lane wrote: > Thomas Munro writes: > > OK. One idea is to provide a WaitLatchUsec(), which is just some > > cross platform donkeywork that I think I know how to type in, and it > > would have to round up on poll() and Windows builds. The

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-09 Thread Thomas Munro
On Fri, Mar 10, 2023 at 1:46 PM Tom Lane wrote: > >> I propose therefore that instead of increasing vacuum_cost_limit, > >> what we ought to be doing is reducing vacuum_cost_delay by a similar > >> factor. And, to provide some daylight for people to reduce it even > >> more, we ought to arrange

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-09 Thread Thomas Munro
Erm, but maybe I'm just looking at this too myopically. Is there really any point in letting people set it to 0.5, if it behaves as if you'd set it to 1 and doubled the cost limit? Isn't it just more confusing? I haven't read the discussion from when fractional delays came in, where I imagine

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-09 Thread Thomas Munro
On Fri, Mar 10, 2023 at 11:37 AM Nathan Bossart wrote: > > On Thu, Mar 09, 2023 at 05:27:08PM -0500, Tom Lane wrote: > > Is it reasonable to assume that all modern platforms can time > > millisecond delays accurately? Ten years ago I'd have suggested > > truncating the delay to a multiple of

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-09 Thread Thomas Munro
On Fri, Mar 10, 2023 at 11:02 AM Tom Lane wrote: > Thomas Munro writes: > > On Fri, Mar 10, 2023 at 10:26 AM Melanie Plageman > > wrote: > >> I think that 4753ef37e0ed undid the work caf626b2c did to support > >> sub-millisecond delays for vacuum and

Re: Sub-millisecond [autovacuum_]vacuum_cost_delay broken

2023-03-09 Thread Thomas Munro
On Fri, Mar 10, 2023 at 10:26 AM Melanie Plageman wrote: > I think that 4753ef37e0ed undid the work caf626b2c did to support > sub-millisecond delays for vacuum and autovacuum. > > After 4753ef37e0ed, vacuum_delay_point()'s local variable msec is a > double which, after being passed to

Cross-database SERIALIZABLE safe snapshots

2023-03-08 Thread Thomas Munro
elsewhere if reconsidering that. [1] https://www.postgresql.org/message-id/flat/17368-98a4f99e8e4b4402%40postgresql.org From 2156a432b3629d1b7f8bf2fe42f641f8845baf48 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 9 Mar 2023 18:12:17 +1300 Subject: [PATCH] Optimize cross-database SERIALIZABLE

pgsql: Fix race in SERIALIZABLE READ ONLY.

2023-03-08 Thread Thomas Munro
Fix race in SERIALIZABLE READ ONLY. Commit bdaabb9b started skipping doomed transactions when building the list of possible conflicts for SERIALIZABLE READ ONLY. That makes sense, because doomed transactions won't commit, but a couple of subtle things broke: 1. If all uncommitted r/w

pgsql: Fix race in SERIALIZABLE READ ONLY.

2023-03-08 Thread Thomas Munro
Fix race in SERIALIZABLE READ ONLY. Commit bdaabb9b started skipping doomed transactions when building the list of possible conflicts for SERIALIZABLE READ ONLY. That makes sense, because doomed transactions won't commit, but a couple of subtle things broke: 1. If all uncommitted r/w

pgsql: Fix race in SERIALIZABLE READ ONLY.

2023-03-08 Thread Thomas Munro
Fix race in SERIALIZABLE READ ONLY. Commit bdaabb9b started skipping doomed transactions when building the list of possible conflicts for SERIALIZABLE READ ONLY. That makes sense, because doomed transactions won't commit, but a couple of subtle things broke: 1. If all uncommitted r/w

pgsql: Fix race in SERIALIZABLE READ ONLY.

2023-03-08 Thread Thomas Munro
Fix race in SERIALIZABLE READ ONLY. Commit bdaabb9b started skipping doomed transactions when building the list of possible conflicts for SERIALIZABLE READ ONLY. That makes sense, because doomed transactions won't commit, but a couple of subtle things broke: 1. If all uncommitted r/w

pgsql: Fix race in SERIALIZABLE READ ONLY.

2023-03-08 Thread Thomas Munro
Fix race in SERIALIZABLE READ ONLY. Commit bdaabb9b started skipping doomed transactions when building the list of possible conflicts for SERIALIZABLE READ ONLY. That makes sense, because doomed transactions won't commit, but a couple of subtle things broke: 1. If all uncommitted r/w

pgsql: Fix race in SERIALIZABLE READ ONLY.

2023-03-08 Thread Thomas Munro
Fix race in SERIALIZABLE READ ONLY. Commit bdaabb9b started skipping doomed transactions when building the list of possible conflicts for SERIALIZABLE READ ONLY. That makes sense, because doomed transactions won't commit, but a couple of subtle things broke: 1. If all uncommitted r/w

Re: allow_in_place_tablespaces vs. pg_basebackup

2023-03-08 Thread Thomas Munro
On Thu, Mar 9, 2023 at 2:58 PM Kyotaro Horiguchi wrote: > At Wed, 8 Mar 2023 16:30:05 -0500, Robert Haas wrote > in > > I'm not sure how messy it's going to be to clean this up. I think each > > tablespace really needs to go into a separate ${TSOID}.tar file, > > because we've got tons of code

Re: Add error functions: erf() and erfc()

2023-03-08 Thread Thomas Munro
On Thu, Mar 9, 2023 at 1:16 PM Dean Rasheed wrote: > On Wed, 8 Mar 2023 at 23:24, Thomas Munro wrote: > > ... But Oracle, and I think > > several other analytics-focused SQL systems, can do it in a very easy > > built-in way. I think to get at that you pr

Re: Add error functions: erf() and erfc()

2023-03-08 Thread Thomas Munro
On Tue, Feb 28, 2023 at 1:54 AM Dean Rasheed wrote: > Now that we have random_normal(), it seems like it would be useful to > add the error functions erf() and erfc(), which I think are > potentially useful to the people who will find random_normal() useful, > and possibly others. > > An

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-03-07 Thread Thomas Munro
On Wed, Mar 8, 2023 at 4:43 PM Anton A. Melnikov wrote: > On 04.03.2023 00:39, Thomas Munro wrote: > > Could we make better use of the safe copy that we have in the log? > > Then the pg_backup_start() subproblem would disappear. Conceptually, > > that'd be just like the wa

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2023-03-06 Thread Thomas Munro
On Tue, Mar 7, 2023 at 7:47 PM Julien Rouhaud wrote: > I registered lapwing as a 32b Debian 7 so I thought it would be expected to > keep it as-is rather than upgrading to all newer major Debian versions, > especially since there were newer debian animal registered (no 32b though > AFAICS).

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2023-03-06 Thread Thomas Munro
On Tue, Mar 7, 2023 at 5:32 PM Michael Paquier wrote: > On Tue, Mar 07, 2023 at 03:44:46PM +1300, Thomas Munro wrote: > > Oh, you already pushed a fix. But now I'm wondering if it's useful to > > have old buggy compilers set to run with -Werror. > > Yes, as far as I can s

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2023-03-06 Thread Thomas Munro
On Tue, Mar 7, 2023 at 3:42 PM Thomas Munro wrote: > Apparently ye olde GCC 4.7 on "lapwing" doesn't like the way you > initialised that struct. I guess it wants {{0}} instead of {0}. > Apparently old GCC was wrong about that warning[1], but that system > doesn't have

Re: Use pg_pwritev_with_retry() instead of write() in dir_open_for_write() to avoid partial writes?

2023-03-06 Thread Thomas Munro
On Mon, Mar 6, 2023 at 5:30 PM Michael Paquier wrote: > On Mon, Feb 20, 2023 at 01:54:00PM +0530, Bharath Rupireddy wrote: > > I ran some tests on my dev system [1] and I don't see much difference > > between v3 and v4. So, +1 for v3 patch (+ argument order swap) from > > Andres to keep the code

pgsql: Fix assert failures in parallel SERIALIZABLE READ ONLY.

2023-03-05 Thread Thomas Munro
Fix assert failures in parallel SERIALIZABLE READ ONLY. 1. Make sure that we don't decrement SxactGlobalXminCount twice when the SXACT_FLAG_RO_SAFE optimization is reached in a parallel query. This could trigger a sanity check failure in assert builds. Non-assert builds recompute the count in

pgsql: Fix assert failures in parallel SERIALIZABLE READ ONLY.

2023-03-05 Thread Thomas Munro
Fix assert failures in parallel SERIALIZABLE READ ONLY. 1. Make sure that we don't decrement SxactGlobalXminCount twice when the SXACT_FLAG_RO_SAFE optimization is reached in a parallel query. This could trigger a sanity check failure in assert builds. Non-assert builds recompute the count in

pgsql: Fix assert failures in parallel SERIALIZABLE READ ONLY.

2023-03-05 Thread Thomas Munro
Fix assert failures in parallel SERIALIZABLE READ ONLY. 1. Make sure that we don't decrement SxactGlobalXminCount twice when the SXACT_FLAG_RO_SAFE optimization is reached in a parallel query. This could trigger a sanity check failure in assert builds. Non-assert builds recompute the count in

pgsql: Fix assert failures in parallel SERIALIZABLE READ ONLY.

2023-03-05 Thread Thomas Munro
Fix assert failures in parallel SERIALIZABLE READ ONLY. 1. Make sure that we don't decrement SxactGlobalXminCount twice when the SXACT_FLAG_RO_SAFE optimization is reached in a parallel query. This could trigger a sanity check failure in assert builds. Non-assert builds recompute the count in

pgsql: Fix assert failures in parallel SERIALIZABLE READ ONLY.

2023-03-05 Thread Thomas Munro
Fix assert failures in parallel SERIALIZABLE READ ONLY. 1. Make sure that we don't decrement SxactGlobalXminCount twice when the SXACT_FLAG_RO_SAFE optimization is reached in a parallel query. This could trigger a sanity check failure in assert builds. Non-assert builds recompute the count in

Re: [PATCH] Add CANONICAL option to xmlserialize

2023-03-05 Thread Thomas Munro
On Mon, Mar 6, 2023 at 11:20 AM Jim Jones wrote: > On 05.03.23 22:00, Thomas Munro wrote: > > could be something to do with > > our environment, since .cirrus.yml sets LANG=C in the 32 bit test run > > -- maybe try that locally? > Also using LANGUAGE=C the result is the

Re: Refactor to introduce pg_strcoll().

2023-03-05 Thread Thomas Munro
+/* Win32 does not have UTF-8, so we need to map to UTF-16 */ I wonder if this is still true. I think in Windows 10+ you can enable UTF-8 support. Then could you use strcoll_l() directly? I struggled to understand that, but I am a simple Unix hobbit from the shire so I dunno. (Perhaps the

Re: [PATCH] Add CANONICAL option to xmlserialize

2023-03-05 Thread Thomas Munro
On Mon, Mar 6, 2023 at 7:44 AM Jim Jones wrote: > The attached version includes documentation and tests to the patch. The CI run for that failed in an interesting way, only on Debian + Meson, 32 bit. The diffs appear to show that psql has a different opinion of the column width, while building

Re: Latches vs lwlock contention

2023-03-04 Thread Thomas Munro
On Sat, Jan 28, 2023 at 3:39 AM vignesh C wrote: > On Tue, 1 Nov 2022 at 16:40, Thomas Munro wrote: > > Here's an attempt at that. There aren't actually any cases of uses of > > this stuff in critical sections here, so perhaps I shouldn't bother > > with that part. The pa

Re: Add standard collation UNICODE

2023-03-04 Thread Thomas Munro
On Sun, Mar 5, 2023 at 7:30 AM Jeff Davis wrote: > Sorting by codepoint should be encoding-independent (i.e. decode to > codepoint first); but the C collation is just strcmp, which is > encoding-dependent. So is UCS_BASIC wrong today? It's created for UTF-8 only, and UTF-8 sorts the same way as

Re: RADIUS tests and improvements

2023-03-03 Thread Thomas Munro
}/t/001_auth.pl. For this one, I just copied the second style, creating src/test/radius/t/001_auth.pl. I can't explain why it should be like that, though. If I propose another test for PAM, where should it go? From d7f57dfe4a316c8ac1270a5f35f837447e335153 Mon Sep 17 00:00:00 2001 From: Thomas Munro

Re: 13.x, stream replication and locale(?) issues

2023-03-03 Thread Thomas Munro
On Wed, Mar 1, 2023 at 10:30 AM Thomas Munro wrote: > On Wed, Mar 1, 2023 at 12:09 AM Eugene M. Zheganin wrote: > > 3) how do I fix it ? Should I take locale sources for ru_RU.utf8 on Linux > > and compile it on FreeBSD - will it help ? > > Out of curiosity (I'm not sa

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-03-03 Thread Thomas Munro
On Fri, Feb 24, 2023 at 11:12 PM Anton A. Melnikov wrote: > On 17.02.2023 06:21, Thomas Munro wrote: > > BTW there are at least two other places where PostgreSQL already knows > > that concurrent reads and writes are possibly non-atomic (and we also > > don't even try to ge

Re: Understanding, testing and improving our Windows filesystem code

2023-03-02 Thread Thomas Munro
On Thu, Jan 5, 2023 at 1:06 AM vignesh C wrote: > On Tue, 25 Oct 2022 at 09:42, Thomas Munro wrote: > > I pushed the bug fixes from this series, without their accompanying > > tests. Here's a rebase of the test suite, with all those tests now > > squashed into the main

Re: File descriptors in exec'd subprocesses

2023-03-02 Thread Thomas Munro
On Thu, Mar 2, 2023 at 9:57 AM Thomas Munro wrote: > On Thu, Mar 2, 2023 at 9:49 AM Gregory Stark (as CFM) > wrote: > > On Mon, 20 Feb 2023 at 23:04, Thomas Munro wrote: > > > Done like that in this version. This is the version I'm thinking of > > > committing,

pgsql: Don't leak descriptors into subprograms.

2023-03-02 Thread Thomas Munro
Don't leak descriptors into subprograms. Open long-lived data and WAL file descriptors with O_CLOEXEC. This flag was introduced by SUSv4 (POSIX.1-2008), and by now all of our target Unix systems have it. Our open() implementation for Windows already had that behavior, so provide a dummy

Re: We shouldn't signal process groups with SIGQUIT

2023-03-01 Thread Thomas Munro
On Thu, Mar 2, 2023 at 1:09 PM Andres Freund wrote: > On 2023-03-02 12:29:28 +1300, Thomas Munro wrote: > > ... Huh... what am I missing? I > > thought the only risk was handlers running in the opposite of send > > order because they 'overlapped', not non-handler code bei

Re: We shouldn't signal process groups with SIGQUIT

2023-03-01 Thread Thomas Munro
On Tue, Feb 28, 2023 at 5:45 PM Michael Paquier wrote: > On Tue, Feb 14, 2023 at 12:47:12PM -0800, Andres Freund wrote: > > Just naively hacking this behaviour change into the current code, would > > yield > > sending SIGQUIT to postgres, and then SIGTERM to the whole process > > group. Which

Re: File descriptors in exec'd subprocesses

2023-03-01 Thread Thomas Munro
On Thu, Mar 2, 2023 at 9:49 AM Gregory Stark (as CFM) wrote: > On Mon, 20 Feb 2023 at 23:04, Thomas Munro wrote: > > > > Done like that in this version. This is the version I'm thinking of > > committing, unless someone wants to argue for another level. > > FWIW t

Re: Interval in hours but not in days Leap second not taken into account

2023-02-28 Thread Thomas Munro
On Mon, Feb 27, 2023 at 8:26 PM PALAYRET Jacques wrote: > # PostgreSQL does not take into account the additional second (leap second) > in some calendar days ; eg. 2016, 31 dec. : > SELECT to_timestamp('20170102 10:11:12','mmdd hh24:mi:ss') - > to_timestamp('20161230 00:00:00','mmdd

Re: 13.x, stream replication and locale(?) issues

2023-02-28 Thread Thomas Munro
On Wed, Mar 1, 2023 at 12:09 AM Eugene M. Zheganin wrote: > 3) how do I fix it ? Should I take locale sources for ru_RU.utf8 on Linux and > compile it on FreeBSD - will it help ? Out of curiosity (I'm not saying it's a good idea!), do you know if FreeBSD's localedef can compile glibc's

Re: stopgap fix for signal handling during restore_command

2023-02-23 Thread Thomas Munro
On Fri, Feb 24, 2023 at 1:25 PM Thomas Munro wrote: > ENOCOFFEE Erm, I realised after sending that I'd accidentally sent a version that uses fork() anyway, and now if I change it back to vfork() it doesn't fail the way I wanted to demonstrate, at least on Linux. I don't have time or des

Re: pgsql: Refactor to add pg_strcoll(), pg_strxfrm(), and variants.

2023-02-23 Thread Thomas Munro
On Fri, Feb 24, 2023 at 1:20 PM Justin Pryzby wrote: > These patches cause warnings under MSVC. > > Of course my patch to improve CI by warning about compiler warnings is > the only one to notice. > > https://cirrus-ci.com/task/6199582053367808 It's a shame that it fails the whole Windows task,

<    4   5   6   7   8   9   10   11   12   13   >