Re: Direct I/O

2023-04-17 Thread Thomas Munro
On Tue, Apr 18, 2023 at 4:06 AM Tom Lane wrote: > Robert Haas writes: > > On Sat, Apr 15, 2023 at 2:19 PM Tom Lane wrote: > >> I get the impression that we are going to need an actual runtime > >> test if we want to defend against this. Not entirely convinced > >> it's worth the trouble. Who,

Re: check_strxfrm_bug()

2023-04-16 Thread Thomas Munro
On Sun, Dec 18, 2022 at 10:27 AM Thomas Munro wrote: > With my garbage collector hat on, that made me wonder if there was > some more potential cleanup here: could we require locale_t yet? The > last straggler systems on our target OS list to add the POSIX locale_t > stuff were

Re: Where are we on supporting LLVM's opaque-pointer changes?

2023-04-15 Thread Thomas Munro
On Sat, Apr 15, 2023 at 2:31 AM Tom Lane wrote: > I know we've been letting this topic slide, but we are out of runway. > I propose adding this as a must-fix open item for PG 16. I had a patch that solved many of the problems[1], but it isn't all the way there and I got stuck. I am going to

Re: Direct I/O

2023-04-15 Thread Thomas Munro
On Sun, Apr 16, 2023 at 6:19 AM Tom Lane wrote: > So apparently, the fact that you even get a warning about the > alignment not being honored is something OpenBSD patched in > after-the-fact; it's not there in genuine vintage gcc. Ah, that is an interesting discovery, and indeed kills the

Re: Direct I/O

2023-04-14 Thread Thomas Munro
On Sat, Apr 15, 2023 at 7:50 AM Mikael Kjellström wrote: > want me to switch to clang instead? I vote +1, that's the system compiler in modern OpenBSD. https://www.cambus.net/the-state-of-toolchains-in-openbsd/ As for curculio, I don't understand the motivation for maintaining that machine.

Re: Direct I/O

2023-04-14 Thread Thomas Munro
On Sat, Apr 15, 2023 at 7:38 AM Tom Lane wrote: > Andres Freund writes: > > On 2023-04-14 15:21:18 -0400, Tom Lane wrote: > >> +1 for that, though. (Also, the fact that these animals aren't > >> actually failing suggests that 004_io_direct.pl needs expansion.) > > > It's skipped, due to lack of

Re: OOM in hash join

2023-04-14 Thread Thomas Munro
On Fri, Apr 14, 2023 at 11:43 PM Jehan-Guillaume de Rorthais wrote: > Would you be able to test the latest patchset posted [1] ? This does not fix > the work_mem overflow, but it helps to keep the number of batches > balanced and acceptable. Any feedback, comment or review would be useful. > >

Re: OOM in hash join

2023-04-14 Thread Thomas Munro
On Fri, Apr 14, 2023 at 10:59 PM Konstantin Knizhnik wrote: > Too small value of work_mem cause memory overflow in parallel hash join > because of too much number batches. Yeah. Not only in parallel hash join, but in any hash join (admittedly parallel hash join has higher per-batch overheads;

Build farm breakage over time

2023-04-13 Thread Thomas Munro
Just for fun, I broke time up into 15 minute intervals and counted how many machines were showing red on HEAD at each sample point (lateral join for last tick interpolation of data I collect from the BF), and plotted that over time. See attached. I excluded seawasp (it tells us about *future*

Re: Wrong results from Parallel Hash Full Join

2023-04-13 Thread Thomas Munro
On Thu, Apr 13, 2023 at 12:31 PM Melanie Plageman wrote: > On Wed, Apr 12, 2023 at 6:50 PM Thomas Munro wrote: > > I think "Discussion:" footers are supposed to use > > https://postgr.es/m/XXX shortened URLs. > > Hmm. Is the problem with mine that I inc

pgsql: Fix PHJ match bit initialization.

2023-04-13 Thread Thomas Munro
Fix PHJ match bit initialization. Hash join tuples reuse the HOT status bit to indicate match status during hash join execution. Correct reuse requires clearing the bit in all tuples. Serial hash join and parallel multi-batch hash join do so upon inserting the tuple into the hashtable. Single

Re: Backends stunk in wait event IPC/MessageQueueInternal

2023-04-13 Thread Thomas Munro
On Sun, Aug 28, 2022 at 11:03 AM Thomas Munro wrote: > On Sun, Jun 26, 2022 at 11:18 AM Thomas Munro wrote: > > On Tue, May 17, 2022 at 3:31 PM Thomas Munro wrote: > > > On Mon, May 16, 2022 at 3:45 PM Japin Li wrote: > > > > Maybe use the _

Re: Direct I/O

2023-04-12 Thread Thomas Munro
Thanks both for looking, and thanks for the explanation Ilmari. Pushed with your improvements. The 4 CI systems run the tests (Windows and Mac by special always-expected-to-work case, Linux and FreeBSD by successful pre-flight perl test of O_DIRECT), and I also tested three unusual systems, two

pgsql: Skip the 004_io_direct.pl test if a pre-flight check fails.

2023-04-12 Thread Thomas Munro
Skip the 004_io_direct.pl test if a pre-flight check fails. The test previously had a list of OSes that direct I/O was expected to work on. That worked well enough for the systems in our build farm, but didn't survive contact with the Debian build bots running on tmpfs via overlayfs. tmpfs does

Re: Clean up hba.c of code freeing regexps

2023-04-12 Thread Thomas Munro
On Thu, Apr 13, 2023 at 12:16 PM Michael Paquier wrote: > The logic in hba.c that scans all the HBA and ident lines to any > regexps can be simplified a lot. Most of this code is new in 16~, so > I think that it is worth cleaning up this stuff now rather than wait > for 17 to open for business.

Re: Wrong results from Parallel Hash Full Join

2023-04-12 Thread Thomas Munro
On Thu, Apr 13, 2023 at 9:48 AM Melanie Plageman wrote: > Attached patch includes the fix for ExecParallelHashTableInsert() as > well as a test. I toyed with adapting one of the existing parallel full > hash join tests to cover this case, however, I think Richard's repro is > much more clear.

Re: Parallel Full Hash Join

2023-04-12 Thread Thomas Munro
On Mon, Apr 10, 2023 at 11:33 AM Michael Paquier wrote: > On Sat, Apr 08, 2023 at 02:19:54PM -0400, Melanie Plageman wrote: > > Another worker attached to the batch barrier, saw that it was in > > PHJ_BATCH_SCAN, marked it done and detached. This is fine. > > BarrierArriveAndDetachExceptLast() is

pgsql: Remove overzealous assertion from PHJ.

2023-04-12 Thread Thomas Munro
Remove overzealous assertion from PHJ. We can't assert that we're the only process attached to a barrier after BarrierArriveAndDetachExceptLast(). Although that'll be true almost always, a late-starting parallel worker can attach very briefly (that is, immediately detach after checking the

Re: Direct I/O

2023-04-12 Thread Thomas Munro
On Wed, Apr 12, 2023 at 5:48 PM Thomas Munro wrote: > On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro wrote: > > On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote: > > > I'm hitting a panic in t_004_io_direct. The build is running on > > > overlayfs on tmpfs/ext4 (upp

Re: Direct I/O

2023-04-11 Thread Thomas Munro
On Wed, Apr 12, 2023 at 3:04 PM Thomas Munro wrote: > On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote: > > I'm hitting a panic in t_004_io_direct. The build is running on > > overlayfs on tmpfs/ext4 (upper/lower) which is probably a weird > > combination but has wor

Re: Direct I/O

2023-04-11 Thread Thomas Munro
On Wed, Apr 12, 2023 at 2:56 PM Christoph Berg wrote: > I'm hitting a panic in t_004_io_direct. The build is running on > overlayfs on tmpfs/ext4 (upper/lower) which is probably a weird > combination but has worked well for building everything over the last > decade. On Debian unstable: > >

Re: v15b1: FailedAssertion("segment_map->header->magic == (DSA_SEGMENT_HEADER_MAGIC ^ area->control->handle ^ index)", File: "dsa.c", ..)

2023-04-11 Thread Thomas Munro
On Wed, Apr 12, 2023 at 11:37 AM Justin Pryzby wrote: > $ ls /dev/shm/ |grep 3696856876 || echo not found > not found Oh, of course it would have restarted after it crashed and unlinked that... So the remaining traces of that memory *might* be in the core file, depending (IIRC) on the core

Re: v15b1: FailedAssertion("segment_map->header->magic == (DSA_SEGMENT_HEADER_MAGIC ^ area->control->handle ^ index)", File: "dsa.c", ..)

2023-04-11 Thread Thomas Munro
On Wed, Apr 12, 2023 at 7:46 AM Justin Pryzby wrote: > Unfortunately: > (gdb) p area->control->handle > $3 = 0 > (gdb) p segment_map->header->magic > value has been optimized out > (gdb) p index > $4 = Hmm, well index I can find from parameters: > #2 0x00991470 in ExceptionalCondition

Re: cfbot is listing committed patches?

2023-04-11 Thread Thomas Munro
On Tue, Apr 11, 2023 at 6:16 PM Peter Smith wrote: > cfbot [1] is listing some already committed patches under the "Needs > Review" category. For example here are some of mine [1][2]. And > because they are already committed, the 'apply' fails, so they get > flagged by cfbot as needed rebase. >

Re: longfin missing gssapi_ext.h

2023-04-10 Thread Thomas Munro
On Tue, Apr 11, 2023 at 2:53 PM Thomas Munro wrote: > On Tue, Apr 11, 2023 at 2:31 AM Stephen Frost wrote: > > Have you tried running the tests in src/test/kerberos with elver? Or is > > it configured to run them? Would be awesome if it could be, or if > > there's issues w

Re: Direct I/O

2023-04-10 Thread Thomas Munro
On Tue, Apr 11, 2023 at 2:31 PM Thomas Munro wrote: > I tried to find out what POSIX says about this (But of course whatever it might say is of especially limited value when O_DIRECT is in the picture, being completely unstandardised. Really I guess all they meant was "if you *copy* s

Re: longfin missing gssapi_ext.h

2023-04-10 Thread Thomas Munro
On Tue, Apr 11, 2023 at 2:31 AM Stephen Frost wrote: > Have you tried running the tests in src/test/kerberos with elver? Or is > it configured to run them? Would be awesome if it could be, or if > there's issues with running the tests on FBSD w/ MIT Kerberos, I'd be > happy to try and help work

Re: Direct I/O

2023-04-10 Thread Thomas Munro
On Tue, Apr 11, 2023 at 2:15 PM Andres Freund wrote: > And the fix has been merged into > https://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git/log/?h=for-next > > I think that means it'll have to wait for 6.4 development to open (in a few > weeks), and then will be merged into the

Re: Direct I/O

2023-04-10 Thread Thomas Munro
On Mon, Apr 10, 2023 at 7:27 PM Thomas Munro wrote: > Debian's 6.0.10-2 kernel (Debian 12 on a random laptop). Realising I hadn't updated for a bit, I did so and it still reproduces on: $ uname -a Linux x1 6.1.0-7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-1 (2023-03-19) x86_64 GNU/Linux

Re: Direct I/O

2023-04-10 Thread Thomas Munro
On Mon, Apr 10, 2023 at 2:57 PM Andres Freund wrote: > Have you tried to write a reproducer for this that doesn't involve postgres? I tried a bit. I'll try harder soon. > ... What kernel version did you repro > this on Thomas? Debian's 6.0.10-2 kernel (Debian 12 on a random laptop). Here's

Re: Direct I/O

2023-04-09 Thread Thomas Munro
On Mon, Apr 10, 2023 at 8:43 AM Tom Lane wrote: > Boy, it's hard to look at that trace and not call it a filesystem bug. Agreed. > Given the apparent dependency on COW, I wonder if this has something > to do with getting confused about which copy is current? Yeah, I suppose it would require

Re: Direct I/O

2023-04-09 Thread Thomas Munro
On Sun, Apr 9, 2023 at 11:25 PM Andrew Dunstan wrote: > Didn't seem to make any difference. Thanks for testing. I think it's COW (and I think that implies also checksums?) that needs to be turned off, at least based on experiments here.

Re: Direct I/O

2023-04-09 Thread Thomas Munro
On Sun, Apr 9, 2023 at 4:52 PM Thomas Munro wrote: > Here, btrfs seems to be taking a different path that I can't quite > make out... I see no warning/error about a checksum failure like [1], > and we apparently managed to read something other than a mix of the > old and new page con

Re: Direct I/O

2023-04-08 Thread Thomas Munro
Indeed, I can't reproduce this with (our) checksums on. I also can't reproduce it with O_DIRECT off. I also can't reproduce it if I use "mkdir pgdata && chattr +C pgdata && initdb -D pgdata" to have a pgdata directory with copy-on-write and (their) checksums disabled. But it reproduces quite

Re: Direct I/O

2023-04-08 Thread Thomas Munro
On Sun, Apr 9, 2023 at 2:18 PM Andres Freund wrote: > On 2023-04-09 13:55:33 +1200, Thomas Munro wrote: > > I think that particular thing might relate to modifications of the > > user buffer while a write is in progress (breaking btrfs's internal > > checksums). I don't th

Re: Direct I/O

2023-04-08 Thread Thomas Munro
On Sun, Apr 9, 2023 at 11:05 AM Tom Lane wrote: > Googling finds a lot of suggestions that O_DIRECT doesn't play nice > with btrfs, for example > > https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg92824.html > > It's not clear to me how much of that lore is still current, > but it's

Re: Direct I/O

2023-04-08 Thread Thomas Munro
On Sun, Apr 9, 2023 at 10:17 AM Andrew Dunstan wrote: > I can run the test in isolation, and it's get an error reliably. Random idea: it looks like you have compression enabled. What if you turn it off in the directory where the test runs? Something like btrfs property set compression ...

Re: longfin missing gssapi_ext.h

2023-04-08 Thread Thomas Munro
On Sun, Apr 9, 2023 at 6:40 AM Tom Lane wrote: > The exact same thing applies to FreeBSD, except that their in-core > Heimdal is ancient (1.5.2). Also, they do have MIT Kerberos > available as a package [1]. I'd been misled by the lack of a hit > on "kerberos", but "krb5" finds it. Our code

Re: Direct I/O

2023-04-08 Thread Thomas Munro
On Sun, Apr 9, 2023 at 10:08 AM Andrew Dunstan wrote: > btrfs Aha!

Re: Direct I/O

2023-04-08 Thread Thomas Munro
On Sun, Apr 9, 2023 at 9:10 AM Tom Lane wrote: > 2023-04-08 16:50:03.177 EDT [2023-04-08 16:50:03 EDT 3257645:3] > 004_io_direct.pl LOG: statement: select count(*) from t1 > 2023-04-08 16:50:03.316 EDT [2023-04-08 16:50:03 EDT 3257646:1] ERROR: > invalid page in block 56 of relation

pgsql: Use higher wal_level for 004_io_direct.pl.

2023-04-08 Thread Thomas Munro
Use higher wal_level for 004_io_direct.pl. The new direct I/O test deliberately uses a very small shared_buffers to force some disk transfers without making the data set large and slow, but ran into a problem with wal_level = minimal: log_newpage_range() pins many buffers, leading to a few

Re: Direct I/O

2023-04-08 Thread Thomas Munro
On Sun, Apr 9, 2023 at 6:55 AM Andres Freund wrote: > Given the frequency of failures on this in the buildfarm, I propose using the > temporary workaround of using wal_level=replica. That avoids the use of the > over-eager log_newpage_range(). Will do.

pgsql: Update tsearch regex memory management.

2023-04-08 Thread Thomas Munro
Update tsearch regex memory management. Now that our regex engine uses palloc(), it's not necessary to set up a special memory context callback to free compiled regexes. The regex has no resources other than the memory that is already going to be freed in bulk. Reviewed-by: Tom Lane

pgsql: Update contrib/trgm_regexp's memory management.

2023-04-08 Thread Thomas Munro
Update contrib/trgm_regexp's memory management. While no code change was necessary for this code to keep working, we don't need to use PG_TRY()/PG_FINALLY() with explicit clean-up while working with regexes anymore. Reviewed-by: Tom Lane Discussion:

pgsql: Use MemoryContext API for regex memory management.

2023-04-08 Thread Thomas Munro
Use MemoryContext API for regex memory management. Previously, regex_t objects' memory was managed with malloc() and free() directly. Switch to palloc()-based memory management instead. Advantages: * memory used by cached regexes is now visible with MemoryContext observability tools *

pgsql: Redesign interrupt/cancel API for regex engine.

2023-04-08 Thread Thomas Munro
Redesign interrupt/cancel API for regex engine. Previously, a PostgreSQL-specific callback checked by the regex engine had a way to trigger a special error code REG_CANCEL if it detected that the next call to CHECK_FOR_INTERRUPTS() would certainly throw via ereport(). A later proposed bugfix

Re: Direct I/O

2023-04-08 Thread Thomas Munro
On Sat, Apr 8, 2023 at 4:59 PM Thomas Munro wrote: > On Sat, Apr 8, 2023 at 4:47 PM Thomas Munro wrote: > > After a bit more copy-editing on docs and comments and a round of > > automated indenting, I have now pushed this. I will now watch the > > build farm. I teste

Re: broken master branch

2023-04-08 Thread Thomas Munro
On Sat, Apr 8, 2023 at 8:04 PM Pavel Stehule wrote: > on fresh Fedora 38, I cannot to run regress tests Looks like the new LLVM 16. I'll try to look at this again next week. In the meantime you could try using 15.

Re: Direct I/O

2023-04-07 Thread Thomas Munro
On Sat, Apr 8, 2023 at 4:47 PM Thomas Munro wrote: > After a bit more copy-editing on docs and comments and a round of > automated indenting, I have now pushed this. I will now watch the > build farm. I tested on quite a few OSes that I have access to, but > this is obvious

Re: Direct I/O

2023-04-07 Thread Thomas Munro
I did some testing with non-default block sizes, and found a few minor things that needed adjustment. The short version is that I blocked some configurations that won't work or would break an assertion. After a bit more copy-editing on docs and comments and a round of automated indenting, I have

pgsql: Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.

2023-04-07 Thread Thomas Munro
() and smgrextend() are correctly aligned, unless PG_O_DIRECT is 0 (= stack alignment tricks may be unavailable) or the block size has been set too small to allow arrays of buffers to be all aligned. Author: Thomas Munro Author: Andres Freund Reviewed-by: Justin Pryzby Discussion: https://postgr.es/m/ca

pgsql: Add io_direct setting (developer-only).

2023-04-07 Thread Thomas Munro
]sync and wal_level=minimal (which also requires max_wal_senders=0). Those are non-default and unlikely settings, and this behavior wasn't (correctly) documented. The same effect can be achieved with io_direct=wal. Author: Thomas Munro Author: Andres Freund Author: Bharath Rupireddy Reviewed

check_GUC_init(wal_writer_flush_after) fails with non-default block size

2023-04-07 Thread Thomas Munro
: 1519, PID: 84605 From 48d971e0b19f770991e334b8dc38422462b4485e Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 8 Apr 2023 13:12:48 +1200 Subject: [PATCH] Fix default wal_writer_flush_after value. Commit a73952b7956 requires default values in guc_table.c and C variable initializers to match.

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2023-04-07 Thread Thomas Munro
en take a little bit longer on the recovery conflict patch itself (v6-0005) on the basis that it's bugfix work and not subject to the feature freeze. From a21a43bf5b1ba073abb3238968b9f8d13b1b318a Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 4 Jan 2023 14:15:40 +1300 Subject: [PATCH v6 1/

Re: Direct I/O

2023-04-07 Thread Thomas Munro
. [1] https://twitter.com/MengTangmu/status/994770040745615361 [2] http://kos.enix.org/pub/gingell8.pdf From c6e01d506762fb7c11a3fb31d56902fa53ea822b Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Tue, 13 Dec 2022 16:25:59 +1300 Subject: [PATCH v4 1/3] Introduce PG_IO_ALIGN_SIZE and align all I/O buff

Re: Using each rel as both outer and inner for JOIN_ANTI

2023-04-06 Thread Thomas Munro
On Thu, Apr 6, 2023 at 6:40 PM Richard Guo wrote: > Seems it wins if the parallel scan becomes part of a hash join in final > plan. I wonder if we have a way to know that in this early stage. I haven't tried but I'm not sure off the top of my head how to make a decision that early unless it's

Re: Using each rel as both outer and inner for JOIN_ANTI

2023-04-05 Thread Thomas Munro
On Thu, Apr 6, 2023 at 12:17 PM Thomas Munro wrote: > I tried the original example from the top of this thread and saw a > decent speedup from parallelism, but only if I set > min_parallel_table_scan_size=0, and otherwise it doesn't choose > Parallel Hash Right Anti Join. Same if I

Re: Using each rel as both outer and inner for JOIN_ANTI

2023-04-05 Thread Thomas Munro
On Thu, Apr 6, 2023 at 9:11 AM Tom Lane wrote: > Richard Guo writes: > > Thanks for reminding. Attached is the rebased patch, with no other > > changes. I think the patch is ready for commit. > > Pushed after a little further fooling with the comments. I also had > to rebase it over 11c2d6fdf

Re: How should we wait for recovery conflict resolution?

2023-04-05 Thread Thomas Munro
On Thu, Apr 6, 2023 at 7:46 AM Thomas Munro wrote: > Initially I was suspicious that there may be tricky races to deal with > around that wakeup logic, and the poll/sleep loop was due to an > inability to come up with something reliable. (Oops lost a sentence) ... but then I realised t

How should we wait for recovery conflict resolution?

2023-04-05 Thread Thomas Munro
277740 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 5 Apr 2023 17:21:18 +1200 Subject: [PATCH] WIP: latchify standby sleep diff --git a/src/backend/storage/ipc/standby.c b/src/backend/storage/ipc/standby.c index 9f56b4e95c..7770877d9b 100644 --- a/src/backend/storage/ipc/standby.c +++ b/s

Re: Check whether binaries can be released for s390x

2023-04-04 Thread Thomas Munro
On Wed, Apr 5, 2023 at 3:03 AM Robert Haas wrote: > On Tue, Apr 4, 2023 at 9:30 AM Namrata Bhave wrote: > > Thank you for getting back. > > > > The request is mainly for the developer community to build and publish > > s390x binaries, apologies if I wasn't clear earlier. > > We can provide

Re: Parallel Full Hash Join

2023-04-04 Thread Thomas Munro
On Wed, Apr 5, 2023 at 7:37 AM Tom Lane wrote: > The comment is no longer in sync with the code: this if-test used to > reject JOIN_FULL and JOIN_RIGHT, and no longer does so, but the comment > still claims it should. Shouldn't we drop the sentence beginning > "Similarly"? (I see that there's

pgsql: Remove comment obsoleted by 11c2d6fd.

2023-04-04 Thread Thomas Munro
Remove comment obsoleted by 11c2d6fd. Reported-by: Tom Lane Discussion: https://postgr.es/m/1604497.1680637072%40sss.pgh.pa.us Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/f303ec6210c4b858fd68531c92b8f0d490c91db5 Modified Files --

Re: ICU locale validation / canonicalization

2023-04-04 Thread Thomas Munro
MSVC now says this on master: [17:48:12.446] c:\cirrus\src\backend\utils\adt\pg_locale.c(2912) : warning C4715: 'icu_language_tag': not all control paths return a value CI doesn't currently fail for MSVC warnings, so it's a bit hidden. FWIW cfbot does show this with a ⚠ sign with its new system

Re: Is RecoveryConflictInterrupt() entirely safe in a signal handler?

2023-04-02 Thread Thomas Munro
On Sat, Jan 14, 2023 at 3:23 PM Thomas Munro wrote: > On Thu, Jan 5, 2023 at 2:14 PM Tom Lane wrote: > > The rcancelrequested API is something that I devised out of whole cloth > > awhile ago. It's not in Tcl's copy of the code, which AFAIK is the > > only other project us

Re: WL_SOCKET_ACCEPT fairness on Windows

2023-03-31 Thread Thomas Munro
tforms' implementations? Done in this version, which is best viewed with git show --ignore-all-space. > Seems like we could use returned_events to get nevents in the way you want it, > without adding even more ++/-- to each of the different events? Yeah. This time I use reported_events. I also fixed

WL_SOCKET_ACCEPT fairness on Windows

2023-03-31 Thread Thomas Munro
it by connecting to two different ports fast enough to exercise the multiple event case. Or maybe latch.c really needs its own test suite. From 733fd375578d8216fed9a81d4a1a575b1f542ca9 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 1 Apr 2023 12:36:12 +1300 Subject: [PATCH] Teach WaitEventSetWait

pgsql: Parallel Hash Full Join.

2023-03-30 Thread Thomas Munro
of the two relations and is either limited by some multiple of work_mem, or it's too big and is partitioned into batches and then the situation is improved by batch-level parallelism. Author: Melanie Plageman Author: Thomas Munro Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/CA%2BhUKG

Re: allow_in_place_tablespaces vs. pg_basebackup

2023-03-28 Thread Thomas Munro
On Tue, Mar 28, 2023 at 5:15 PM Thomas Munro wrote: > I guess we should add a test of -Fp too, to keep it working? Oops, that was already tested in existing tests, so I take that back.

Re: allow_in_place_tablespaces vs. pg_basebackup

2023-03-27 Thread Thomas Munro
no-sync', '-cfast'); + +# Set up an instance. +my $node = PostgreSQL::Test::Cluster->new('main'); +$node->init('allows_streaming' => 1); +$node->start(); + +# Create an in-place tablespace. +$node->safe_psql('postgres', <command_ok( + [ @pg_basebackup_defs, '-D', $backupdir, '-Ft'

Re: Parallel Full Hash Join

2023-03-27 Thread Thomas Munro
ent problem and we don't know what we'll eventually settle on with more research. g.pdf Description: Adobe PDF document From 0bd32dd1a7e45e95a62f7f587bfba64bed87da28 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 24 Mar 2023 14:19:07 +1300 Subject: [PATCH v13 1/4] Scan for unmatched

Re: Parallel Full Hash Join

2023-03-24 Thread Thomas Munro
st, more soon. From 8b526377eb4a4685628624e75743aedf37dd5bfe Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 24 Mar 2023 14:19:07 +1300 Subject: [PATCH v12 1/2] Scan for unmatched hash join tuples in memory order. In a full/right outer join, we need to scan every tuple in the hash table to find the ones that were not matche

Re: cfbot stuck

2023-03-24 Thread Thomas Munro
On Sat, Mar 25, 2023 at 8:23 AM Justin Pryzby wrote: > It looks like cfbot is stuck since 13h ago. > > https://cirrus-ci.com/github/postgresql-cfbot/postgresql https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/ broke its ability to push branches. It's back in business now.

Re: Orphaned wait event

2023-03-23 Thread Thomas Munro
g like that? I suspect that downstream/monitoring tools might appreciate the existence of such a file too. From 13e8d3d10d773a07c5586c6f8efc4c526b5db1d0 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri, 24 Mar 2023 10:47:36 +1300 Subject: [PATCH] Fix missing WAIT_EVENT_SLRU_FLUSH_SYNC reportin

Orphaned wait event

2023-03-22 Thread Thomas Munro
/include/utils/wait_event.h | grep -v src/backend/utils/activity/wait_event.c | grep $X > /dev/null ) then echo "$X is not used" fi done From 5bd5d5448b1fbd01833069af72bf12d19eef42dd Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 23 Mar 2023 14:36:27

Re: refactoring basebackup.c

2023-03-22 Thread Thomas Munro
On Thu, Mar 23, 2023 at 2:50 PM Thomas Munro wrote: > In rem: commit 3500ccc3, > > for X in ` grep -E '^[^*]+event_name = "' > src/backend/utils/activity/wait_event.c | >sed 's/^.* = "//;s/";$//;/unknown/d' ` > do > if ! git grep "$X&

pgsql: Improve the naming of Parallel Hash Join phases.

2023-03-22 Thread Thomas Munro
Improve the naming of Parallel Hash Join phases. * Commit 3048898e dropped -ING from PHJ wait event names. Update the corresponding barrier phases names to match. * Rename the "DONE" phases to "FREE". That's symmetrical with "ALLOCATE", and names the activity that actually happens in that

Re: Commitfest 2023-03 starting tomorrow!

2023-03-21 Thread Thomas Munro
address --++-- 8d578b9b2e37a4d9d6f422ced5126acec62365a7 | 2023-03-21 14:29:34+13 | Melanie Plageman 8d578b9b2e37a4d9d6f422ced5126acec62365a7 | 2023-03-21 14:29:34+13 | Thomas Munro (2 rows) You can really only go back about 5-7 years befor

Re: Using AF_UNIX sockets always for tests on Windows

2023-03-21 Thread Thomas Munro
on feedback already given, not included in this revision. From e7fac7a15ed0eda6516e7fa0917c06e005341b00 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 7 Sep 2022 07:35:11 +1200 Subject: [PATCH v3 1/2] Make mkdtemp() more secure on Windows. MIME-Version: 1.0 Content-Type: text/plain; chars

Re: Assertion failure with barriers in parallel hash join

2023-03-20 Thread Thomas Munro
Pushed and back-patched, with minor comment tweaks. Apologies for taking so long.

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

pgsql: Fix race in parallel hash join batch cleanup, take II.

2023-03-20 Thread Thomas Munro
still negotiate who is cleaning up. Revealed by build farm failures, where BarrierAttach() failed a sanity check assertion, because the memory had been clobbered by dsa_free(). In non-assert builds, the result could be a segmentation fault. Back-patch to all supported releases. Author: Thomas

Re: Commitfest 2023-03 starting tomorrow!

2023-03-20 Thread Thomas Munro
On Tue, Mar 21, 2023 at 9:22 AM Greg Stark wrote: > Yeah, even aside from flappers there's the problem that it's about as > common for real commits to break some test as it is for patches to > start failing tests. So often it's a real failure but it's nothing to > do with the patch, it has to do

Re: Commitfest 2023-03 starting tomorrow!

2023-03-20 Thread Thomas Munro
On Tue, Mar 21, 2023 at 3:15 AM Greg Stark wrote: > The next level of this would be something like notifying the committer > with a list of patches in the CF that a commit broke. I don't > immediately see how to integrate that with our workflow but I have > seen something like this work well in a

Re: Commitfest 2023-03 starting tomorrow!

2023-03-19 Thread Thomas Munro
On Mon, Mar 20, 2023 at 1:10 PM Tom Lane wrote: > Thomas Munro writes: > > ... I suppose then someone might complain > > that it should be clearer if a patch hasn't applied for a very long > > time; suggestions for how to show that are welcome. > > Can you make t

Re: Commitfest 2023-03 starting tomorrow!

2023-03-19 Thread Thomas Munro
On Mon, Mar 20, 2023 at 11:13 AM Thomas Munro wrote: > I realised that part of Alvaro's complaint was probably caused by > cfbot's refusal to show any useful information just because it > couldn't apply a patch the last time it tried. A small improvement > today: now it show

Re: Commitfest 2023-03 starting tomorrow!

2023-03-19 Thread Thomas Munro
On Sun, Mar 19, 2023 at 12:44 PM Justin Pryzby wrote: > On Sat, Mar 18, 2023 at 04:28:02PM -0700, Peter Geoghegan wrote: > > On Sat, Mar 18, 2023 at 4:19 PM Justin Pryzby wrote: > > > The only issue with this is that cfbot has squished all the commits into > > > one, and lost the original commit

Re: File descriptors in exec'd subprocesses

2023-03-17 Thread Thomas Munro
I pushed the libpq changes. I'll leave the pipe2 and accept4 changes on ice for now, maybe for a later cycle (unlike the committed patches, they don't currently fix a known problem, they just avoid some syscalls that are already fairly rare). For the libpq change, the build farm seems happy so

Re: src/test/examples/testlibpq2.c where the HAVE_SYS_SELECT_H is defined.

2023-03-17 Thread Thomas Munro
On Fri, Mar 17, 2023 at 7:48 PM jian he wrote: > Hi, > playing around with $[0] testlibpq2.c example. I wondered where > HAVE_SYS_SELECT_H is defined? > > I searched on the internet, founded that people also asked the same question > in $[1]. > > In my machine, I do have . > system version:

pgsql: libpq: Use modern socket flags, if available.

2023-03-17 Thread Thomas Munro
libpq: Use modern socket flags, if available. Since commit 7627b91cd5d, libpq has used FD_CLOEXEC so that sockets wouldn't be leaked to subprograms. With enough bad luck, a multi-threaded program might fork in between the socket() and fcntl() calls. We can close that tiny gap by using

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

pgsql: Small tidyup for commit d41a178b, part II.

2023-03-16 Thread Thomas Munro
Small tidyup for commit d41a178b, part II. Further to commit 6a9229da, checking for NULL is now redundant. An "out of memory" error would have been thrown already by palloc() and treated as FATAL, so we can delete a few more lines. Back-patch to all releases, like those other commits.

<    3   4   5   6   7   8   9   10   11   12   >