pgsql: Avoid unnecessary "touch meson.build" in vpath builds
Avoid unnecessary "touch meson.build" in vpath builds In e6927270cd1 I added a 'touch meson.build' to configure.ac, to ensure conflicts between in-tree configure based builds and meson builds are automatically detected. Unfortunately I omitted spaces around the condition restricting this to in-tree builds, leading to touch meson.build to also be executed in vpath builds. While the only consequence of this buglet is an unnecessary empty file in build directories, it seems worth backpatching. Reported-by: Christoph Berg Discussion: https://postgr.es/m/20240417230002.mb2gv3hyetyn6...@awork3.anarazel.de Backpatch: 16-, where the meson based build was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/6d4bc96eb77bca6277880f218cd7a2f9e44bddc0 Modified Files -- configure| 2 +- configure.ac | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
pgsql: Avoid unnecessary "touch meson.build" in vpath builds
Avoid unnecessary "touch meson.build" in vpath builds In e6927270cd1 I added a 'touch meson.build' to configure.ac, to ensure conflicts between in-tree configure based builds and meson builds are automatically detected. Unfortunately I omitted spaces around the condition restricting this to in-tree builds, leading to touch meson.build to also be executed in vpath builds. While the only consequence of this buglet is an unnecessary empty file in build directories, it seems worth backpatching. Reported-by: Christoph Berg Discussion: https://postgr.es/m/20240417230002.mb2gv3hyetyn6...@awork3.anarazel.de Backpatch: 16-, where the meson based build was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/2209c70c652e57314961a25725266821721415b8 Modified Files -- configure| 2 +- configure.ac | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
pgsql: Remove GlobalVisTestNonRemovable[Full]Horizon, not used anymore
Remove GlobalVisTestNonRemovable[Full]Horizon, not used anymore GlobalVisTestNonRemovableHorizon() was only used for the implementation of snapshot_too_old, which was removed in f691f5b80a8. As using GlobalVisTestNonRemovableHorizon() is not particularly efficient, no new uses for it should be added. Therefore remove. Discussion: https://postgr.es/m/20240415185720.q4dg4dlcyvvra...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/3ab8cf92754d5ee4390af4e4840fb45d5bc25007 Modified Files -- src/backend/storage/ipc/procarray.c | 30 -- src/include/utils/snapmgr.h | 2 -- 2 files changed, 32 deletions(-)
Re: pgsql: Teach radix tree to embed values at runtime
Hi, On 2024-04-08 11:57:01 +, John Naylor wrote: > Teach radix tree to embed values at runtime > > Previously, the decision to store values in leaves or within the child > pointer was made at compile time, with variable length values using > leaves by necessity. This commit allows introspecting the length of > variable length values at runtime for that decision. This requires > the ability to tell whether the last-level child pointer is actually > a value, so we use a pointer tag in the lowest level bit. > > Use this in TID store. This entails adding a byte to the header to > reserve space for the tag. Commit f35bd9bf3 stores up to three offsets > within the header with no bitmap, and now the header can be embedded > as above. This reduces worst-case memory usage when TIDs are sparse. This isn't quite C99 conformant, and thus breaks on the buildfarm animal set up to test that: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=mylodon=2024-04-08%2012%3A07%3A01 You can't have unnamed structs in C99, that's a C11 feature. I wish we'd move to C11, but ... Greetings, Andres
pgsql: Remove references to old function name
Remove references to old function name In a97bbe1f1df I accidentally referenced heapgetpage(), both in a function name and a comment. But since 44086b09753 the relevant function is named heap_prepare_pagescan(). Rename the new function to page_collect_tuples(). Reported-by: Melanie Plageman Reported-by: David Rowley Discussion: https://postgr.es/m/20240407172615.cocrsvboqm3tt...@awork3.anarazel.de Discussion: https://postgr.es/m/CAApHDvp4SniHopTrVeKWcEvNXFtdki0utAvO=5r7h6tnhtu...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/e3b69be9515979c546fbfc51f9f801c7b198 Modified Files -- src/backend/access/heap/heapam.c | 16 1 file changed, 8 insertions(+), 8 deletions(-)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/4b179a47242993fe48b1bfa21b17dc9d5e2b35a2 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/974374dcd394c80f37e20b692b7d8b122aa8ed33 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/25ee58e8f26df9ce3037258206eb100a3fc55103 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/dcb7cf945c88eabce7f8350023b8184266d96a12 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/af7e90a2778625d63add69165fb68c370509a199 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
pgsql: simplehash: Free collisions array in SH_STAT
simplehash: Free collisions array in SH_STAT While SH_STAT() is only used for debugging, the allocated array can be large, and therefore should be freed. It's unclear why coverity started warning now. Reported-by: Tom Lane Reported-by: Coverity Discussion: https://postgr.es/m/3005248.1712538...@sss.pgh.pa.us Backpatch: 12- Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/eabf98e949e3a655494a32ceb8d146117cce0505 Modified Files -- src/include/lib/simplehash.h | 3 +++ 1 file changed, 3 insertions(+)
Re: pgsql: Send ALPN in TLS handshake, require it in direct SSL connections
Hi, On 2024-04-08 01:25:40 +, Heikki Linnakangas wrote: > Send ALPN in TLS handshake, require it in direct SSL connections > > libpq now always tries to send ALPN. With the traditional negotiated > SSL connections, the server accepts the ALPN, and refuses the > connection if it's not what we expect, but connecting without ALPN is > still OK. With the new direct SSL connections, ALPN is mandatory. > > NOTE: This uses "TBD-pgsql" as the protocol ID. We must register a > proper one with IANA before the release! > > Author: Greg Stark, Heikki Linnakangas > Reviewed-by: Matthias van de Meent, Jacob Champion My compiler complains: [396/992 42 39%] Compiling C object src/backend/postgres_lib.a.p/libpq_be-secure-openssl.c.o ../../../../../home/andres/src/postgresql/src/backend/libpq/be-secure-openssl.c: In function 'alpn_cb': ../../../../../home/andres/src/postgresql/src/backend/libpq/be-secure-openssl.c:1327:69: warning: ordered comparison of pointer with integer zero [-Wextra] 1327 | if (*out == NULL || *outlen > sizeof(alpn_protos) || outlen <= 0) | ^~ [991/992 1 99%] Linking target src/bin/pg_dump/pg_dump And I think it may show why the warning is a good idea - I assume "*outlen <= 0" was intended? Greetings, Andres Freund
Re: pgsql: Use bump memory context for tuplesorts
Hi,
On 2024-04-07 13:52:28 -0400, Tom Lane wrote:
> Melanie Plageman writes:
> > Looks like this assert is tripping on grison [1].
>
> > running bootstrap script ... TRAP: failed Assert("total_allocated ==
> > context->mem_allocated"), File: "bump.c", Line: 808, PID: 30248
>
> The same on mamba. However, I failed to duplicate it in an x86
> (32-bit) VM. I think the critical factor on the failing machines
> is that MAXALIGN is 8 which is more than their sizeof(pointer).
For posterity: I think the above is correct, and that I found the concrete
reason this causes issues on the thread related to the patch. I posted there
because I figured it out after looking at Tomas' gdb output:
https://postgr.es/m/20240407210924.fvwfwbzms3bvtfcd%40awork3.anarazel.de
- Andres
Re: pgsql: Reduce branches in heapgetpage()'s per-tuple loop
Hi, On 2024-04-07 09:14:57 -0400, Melanie Plageman wrote: > On Sun, Apr 7, 2024 at 3:29 AM Andres Freund wrote: > > > > Reduce branches in heapgetpage()'s per-tuple loop > > > > Until now, heapgetpage()'s loop over all tuples performed some conditional > > checks for each tuple, even though condition did not change across the loop. > > I haven't reviewed exactly what this does, but heapgetpage() was > removed (or split into two functions, rather) in 44086b097537, so I > find the naming of this function, or, at least the comments, commit > message and description a bit confusing. Oops. That name was so ingrained in my head that despite looking over these lines quite a few times, I couldn't spot that. Will send an update to the thread. Greetings, Andres Freund
pgsql: Reduce branches in heapgetpage()'s per-tuple loop
Reduce branches in heapgetpage()'s per-tuple loop Until now, heapgetpage()'s loop over all tuples performed some conditional checks for each tuple, even though condition did not change across the loop. This commit fixes that by moving the loop into an inline function. By calling it with different constant arguments, the compiler can generate an optimized loop for the different conditions, at the price of two per-page checks. For cases of all-visible tables and an isolation level other than serializable, speedups of up to 25% have been measured. Reviewed-by: John Naylor Reviewed-by: Zhang Mingli Tested-by: Quan Zongliang Discussion: https://postgr.es/m/20230716015656.xjvemfbp5fysj...@awork3.anarazel.de Discussion: https://postgr.es/m/2ef7ff1b-3d18-2283-61b1-bbd25fc6c...@yeah.net Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a97bbe1f1df9eba0b18207c321c67de80b33db16 Modified Files -- src/backend/access/heap/heapam.c | 102 --- 1 file changed, 74 insertions(+), 28 deletions(-)
pgsql: Fix headerscheck violation introduced in f8ce4ed78ca
Fix headerscheck violation introduced in f8ce4ed78ca Per ci. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/9e7386924e837aef8d48895cf72a6a0b7f78cbe9 Modified Files -- src/bin/pg_combinebackup/reconstruct.h | 1 + 1 file changed, 1 insertion(+)
pgsql: Avoid edge case in pg_visibility test with small shared_buffers
Avoid edge case in pg_visibility test with small shared_buffers Since 82a4edabd27 we can bulk extend relations. The bulk relation extension logic has a heuristic component. Normally the heurstic does not trigger in the occasionally-failing test case, as the relation is only extended once. But with very small shared_buffers the limits for the number of buffers pinned at once prevent the extension from happening at once. With the second "bulk" extension, the heuristic kicks in, and the relation ends up one block bigger. That's ok from a correctness perspective, but changes the results of the test query due to one additional block. We discussed a few more expansive fixes, but for now have decided to avoid this by making the table a bit smaller. Author: Heikki Linnakangas Reported-by: Discussion: https://postgr.es/m/29c74104-210b-ef39-2522-27a6aa7a7...@iki.fi Discussion: https://postgr.es/m/2023091611.2ugpkkkp7bpp4...@awork3.anarazel.de Backpatch: 16-, where the new relation extension logic was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/408209d6a9ae90a822edd48ce3c173b8b7e04ab7 Modified Files -- contrib/pg_visibility/expected/pg_visibility.out | 3 +-- contrib/pg_visibility/sql/pg_visibility.sql | 6 -- 2 files changed, 1 insertion(+), 8 deletions(-)
pgsql: Avoid edge case in pg_visibility test with small shared_buffers
Avoid edge case in pg_visibility test with small shared_buffers Since 82a4edabd27 we can bulk extend relations. The bulk relation extension logic has a heuristic component. Normally the heurstic does not trigger in the occasionally-failing test case, as the relation is only extended once. But with very small shared_buffers the limits for the number of buffers pinned at once prevent the extension from happening at once. With the second "bulk" extension, the heuristic kicks in, and the relation ends up one block bigger. That's ok from a correctness perspective, but changes the results of the test query due to one additional block. We discussed a few more expansive fixes, but for now have decided to avoid this by making the table a bit smaller. Author: Heikki Linnakangas Reported-by: Discussion: https://postgr.es/m/29c74104-210b-ef39-2522-27a6aa7a7...@iki.fi Discussion: https://postgr.es/m/2023091611.2ugpkkkp7bpp4...@awork3.anarazel.de Backpatch: 16-, where the new relation extension logic was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/3a4837fc809a8656374959049f3ac7a09a711334 Modified Files -- contrib/pg_visibility/expected/pg_visibility.out | 3 +-- contrib/pg_visibility/sql/pg_visibility.sql | 6 -- 2 files changed, 1 insertion(+), 8 deletions(-)
pgsql: ci: macos: Choose python version
ci: macos: Choose python version The CI base image used to have a python3 with headers etc installed in PATH, but doesn't anymore. Instead of relying on a specific version in the base image, explicitly install one ourselves. On 16 and HEAD this lead to a build without python support, but on 15 CI failed, due to explicitly enabled python3 support. Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/90677b52126af98d40676cd379b56b52e1428188 Modified Files -- .cirrus.tasks.yml | 3 +++ 1 file changed, 3 insertions(+)
pgsql: ci: macos: Choose python version
ci: macos: Choose python version The CI base image used to have a python3 with headers etc installed in PATH, but doesn't anymore. Instead of relying on a specific version in the base image, explicitly install one ourselves. On 16 and HEAD this lead to a build without python support, but on 15 CI failed, due to explicitly enabled python3 support. Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/897efe0f39bee8ced11871ac65cc8d28d9b8ea1e Modified Files -- .cirrus.tasks.yml | 3 +++ 1 file changed, 3 insertions(+)
pgsql: ci: macos: Choose python version
ci: macos: Choose python version The CI base image used to have a python3 with headers etc installed in PATH, but doesn't anymore. Instead of relying on a specific version in the base image, explicitly install one ourselves. On 16 and HEAD this lead to a build without python support, but on 15 CI failed, due to explicitly enabled python3 support. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/c6f09e188bbc5df8fd96682ba19e96f19d137eb2 Modified Files -- .cirrus.tasks.yml | 3 +++ 1 file changed, 3 insertions(+)
pgsql: meson: macos: Avoid warnings on Sonoma
meson: macos: Avoid warnings on Sonoma Starting with the Sonoma toolchain macos' linker emits warnings when the same library is linked to twice. That's ill considered, as the same library can be used by multiple subsidiary libraries. Luckily there's a flag to suppress that warning. On Ventura meson's default of -Wl,-undefined,dynamic_lookup caused warnings, which we suppressed with -Wl,-undefined,error. Unfortunately that causes a warning on Sonoma, which is absurd, as it's documented linker default. To avoid that warning, only add -Wl,-undefined,error if it does not trigger warnings. Luckily dynamic_lookup doesn't trigger a warning on Sonoma anymore. Discussion: https://postgr.es/m/20231201040515.p5bshhhtfru7d...@awork3.anarazel.de Backpatch: 16-, where the meson build was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/9158e4b9eb9916fa586feb26a6149b59566b78e1 Modified Files -- meson.build | 14 -- 1 file changed, 12 insertions(+), 2 deletions(-)
pgsql: meson: macos: Avoid warnings on Sonoma
meson: macos: Avoid warnings on Sonoma Starting with the Sonoma toolchain macos' linker emits warnings when the same library is linked to twice. That's ill considered, as the same library can be used by multiple subsidiary libraries. Luckily there's a flag to suppress that warning. On Ventura meson's default of -Wl,-undefined,dynamic_lookup caused warnings, which we suppressed with -Wl,-undefined,error. Unfortunately that causes a warning on Sonoma, which is absurd, as it's documented linker default. To avoid that warning, only add -Wl,-undefined,error if it does not trigger warnings. Luckily dynamic_lookup doesn't trigger a warning on Sonoma anymore. Discussion: https://postgr.es/m/20231201040515.p5bshhhtfru7d...@awork3.anarazel.de Backpatch: 16-, where the meson build was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a3da95deee38ee067b0bead639c830eacbe894d5 Modified Files -- meson.build | 14 -- 1 file changed, 12 insertions(+), 2 deletions(-)
Re: pgsql: meson: docs: Add {html,man} targets, rename install-doc-*
Hi,
On 2023-12-01 09:04:19 -0500, Andrew Dunstan wrote:
> On 2023-11-29 We 07:20, Andrew Dunstan wrote:
> > On 2023-11-28 Tu 21:28, Andres Freund wrote:
> > > On 2023-11-23 08:32:21 -0500, Andrew Dunstan wrote:
> > > > On 2023-11-20 Mo 20:53, Andres Freund wrote:
> > > > > meson: docs: Add {html,man} targets, rename install-doc-*
> > > > >
> > > > > We have toplevel html, man targets in the autoconf build as
> > > > > well. It'd be odd
> > > > > to have an 'html' target but have the install target be
> > > > > 'install-doc-html',
> > > > > thus rename the install targets to match.
> > > >
> > > > This commit of one of its nearby friends appears to have broken
> > > > crake's docs
> > > > build:
> > > >
> > > > ERROR: Can't invoke target `html`: ambiguous name.Add target
> > > > type and/or path:
> > > > - ./doc/src/sgml/html:custom
> > > > - ./doc/src/sgml/html:alias
> > > >
> > > > See<https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake=2023-11-23%2012%3A52%3A04>
> > > >
> > > Ah, I realize now that this is from meson compile html, not 'ninja
> > > html'. That
> > > explains why I couldn't reproduce this initially and why CI didn't
> > > complain.
> > > I don't really understand why meson compile complains in this case.
> > > I assume
> > > you don't want to disambiguate as suggested, by building html:alias
> > > instead?
> > >
> >
> > I've done that as a temporary fix to get crake out of the hole, but it's
> > pretty ugly, and I don't want to do it in a release if at all possible.
>
>
> and doing this has broken the docs build for release 16.
If I can get somebody to comment on
https://postgr.es/m/20231129183619.3hrnwaexbrpygbxg%40awork3.anarazel.de
we can remove the need for the :$buildtype suffix.
Greetings,
Andres Freund
pgsql: meson: Stop using deprecated way getting path of files
meson: Stop using deprecated way getting path of files The just released meson 1.3 strongly deprecated a hack we were using, emitting a noisy warning (the hack basically depended on an implementation detail to work). Turns out there has been a better way available for a while, I just hadn't found it. 1.4 added a more convenient approach, but we can't rely on that. Reviewed-by: Tristan Partin Discussion: https://postgr.es/m/20231129185053.s6c7f73eg7b4z...@awork3.anarazel.de Backpatch: 16-, where the meson build was added. Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/70516b8fc3ee77113013930efb7bfdbabcfed885 Modified Files -- meson.build | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-)
pgsql: meson: Stop using deprecated way getting path of files
meson: Stop using deprecated way getting path of files The just released meson 1.3 strongly deprecated a hack we were using, emitting a noisy warning (the hack basically depended on an implementation detail to work). Turns out there has been a better way available for a while, I just hadn't found it. 1.4 added a more convenient approach, but we can't rely on that. Reviewed-by: Tristan Partin Discussion: https://postgr.es/m/20231129185053.s6c7f73eg7b4z...@awork3.anarazel.de Backpatch: 16-, where the meson build was added. Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/aa11a9c14902b7bcb1d73533a0403dc50361112a Modified Files -- meson.build | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-)
Re: pgsql: meson: docs: Add {html,man} targets, rename install-doc-*
Hi, On 2023-11-29 10:05:26 -0500, Andrew Dunstan wrote: > On 2023-11-29 We 08:49, Tom Lane wrote: > > Andrew Dunstan writes: > > > On 2023-11-28 Tu 21:28, Andres Freund wrote: > > > > I don't really understand why meson compile complains in this case. I > > > > assume > > > > you don't want to disambiguate as suggested, by building html:alias > > > > instead? > > > I've done that as a temporary fix to get crake out of the hole, but it's > > > pretty ugly, and I don't want to do it in a release if at all possible. > > Our documentation says specifically that "ninja html" will build the > > HTML format. I would expect that to work by analogy with the "make" > > target; having to spell it differently seems like clearly a bug. > > > > > > > "ninja html" does in fact work. What's not working is "meson compile html". > And it looks like the reason I used that in the buildfarm code is that ninja > doesn't know about other targets like "postgres-US.pdf". It does: ninja help|grep pdf doc/src/sgml/postgres-A4.pdf Build documentation in PDF format, with A4 pages doc/src/sgml/postgres-US.pdf Build documentation in PDF format, with US letter pages "ninja doc/src/sgml/postgres-US.pdf" works and has worked since day one. FWIW, you can continue to use meson compile, you just need to disambiguate the target name: meson compile html:alias Which isn't particularly pretty, but does work. Greetings, Andres Freund
Re: pgsql: meson: docs: Add {html,man} targets, rename install-doc-*
Hi,
On 2023-11-23 08:32:21 -0500, Andrew Dunstan wrote:
> On 2023-11-20 Mo 20:53, Andres Freund wrote:
> > meson: docs: Add {html,man} targets, rename install-doc-*
> >
> > We have toplevel html, man targets in the autoconf build as well. It'd be
> > odd
> > to have an 'html' target but have the install target be 'install-doc-html',
> > thus rename the install targets to match.
>
>
> This commit of one of its nearby friends appears to have broken crake's docs
> build:
>
> ERROR: Can't invoke target `html`: ambiguous name.Add target type and/or path:
> - ./doc/src/sgml/html:custom
> - ./doc/src/sgml/html:alias
>
> See<https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake=2023-11-23%2012%3A52%3A04>
Ah, I realize now that this is from meson compile html, not 'ninja html'. That
explains why I couldn't reproduce this initially and why CI didn't complain.
I don't really understand why meson compile complains in this case. I assume
you don't want to disambiguate as suggested, by building html:alias instead?
Greetings,
Andres Freund
pgsql: meson: Document build targets, add 'help' target
meson: Document build targets, add 'help' target Currently important build targets are somewhat hard to discover. This commit documents important meson build targets in the sgml documentation. But it's awkward to have to lookup build targets in the docs when hacking, so this also adds a 'help' target, printing out the same information. To avoid having to duplicate information in two places, generate both docbook and interactive docs from a single source. Reviewed-by: Peter Eisentraut Discussion: https://postgr.es/m/20231108232121.ww542mt6lfo6f...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/07cb29737a4ea9bd28e436eecb4016c3202e3928 Modified Files -- doc/src/sgml/Makefile | 5 ++- doc/src/sgml/docguide.sgml | 13 --- doc/src/sgml/filelist.sgml | 1 + doc/src/sgml/generate-targets-meson.pl | 63 ++ doc/src/sgml/installation.sgml | 15 doc/src/sgml/meson.build | 9 + doc/src/sgml/targets-meson.txt | 43 +++ meson.build| 7 8 files changed, 148 insertions(+), 8 deletions(-)
pgsql: docs: meson: Change what 'docs' target builds
docs: meson: Change what 'docs' target builds This undoes the change in what the 'docs' target builds 969509c3f2e. Tom was concerned with having a target to just build the html docs, which a prior commit now provided explicitly. A subsequent commit will overhaul the documentation for the documentation targets. While at it, move all target in doc/src/sgml/Makefile up to just after the default "html" target, and add a comment explaining "all" is *not* the default target. Reviewed-by: Peter Eisentraut Discussion: https://postgr.es/m/20230209203855.njrepiupc3rme...@awork3.anarazel.de Discussion: https://postgr.es/m/20231103163848.26egkh5qdgw3v...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/9e5b2a091fd0d672b7b2ff8afcfc9e9a6e9a5ac8 Modified Files -- doc/src/sgml/Makefile | 4 ++-- doc/src/sgml/docguide.sgml | 2 +- doc/src/sgml/meson.build | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
pgsql: meson: Add 'world' target
meson: Add 'world' target We have this for make as well. Reviewed-by: Peter Eisentraut Discussion: https://postgr.es/m/20231103163848.26egkh5qdgw3v...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/6614cfb43c8281feef5c4563ddc36ab04914014e Modified Files -- meson.build | 2 ++ 1 file changed, 2 insertions(+)
pgsql: meson: docs: Add {html,man} targets, rename install-doc-*
meson: docs: Add {html,man} targets, rename install-doc-*
We have toplevel html, man targets in the autoconf build as well. It'd be odd
to have an 'html' target but have the install target be 'install-doc-html',
thus rename the install targets to match.
Reviewed-by: Christoph Berg
Reviewed-by: Peter Eisentraut
Discussion:
https://postgr.es/m/20231103163848.26egkh5qdgw3v...@awork3.anarazel.de
Branch
--
master
Details
---
https://git.postgresql.org/pg/commitdiff/ddcab2a0329511e8872b62f2c77e5fa33547c277
Modified Files
--
doc/src/sgml/meson.build | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
pgsql: meson: Fix missing dependency from install-quiet to sepgsql.sql
meson: Fix missing dependency from install-quiet to sepgsql.sql This could lead to an error like ERROR: File 'contrib/sepgsql/sepgsql.sql' could not be found Backpatch: 16-, where meson was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/823eb3db1c50a6b8a89ebedc1db96b14de140183 Modified Files -- contrib/sepgsql/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: meson: Fix missing dependency from install-quiet to sepgsql.sql
meson: Fix missing dependency from install-quiet to sepgsql.sql This could lead to an error like ERROR: File 'contrib/sepgsql/sepgsql.sql' could not be found Backpatch: 16-, where meson was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/be08a359db7231f81f8d64975f4babf6dd24bfec Modified Files -- contrib/sepgsql/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Re: pgsql: docs: Document --with-selinux/-Dselinux options centrally
Hi, On 2023-11-17 13:23:54 -0800, Andres Freund wrote: > On 2023-11-17 15:49:52 -0500, Tom Lane wrote: > > Andres Freund writes: > > > docs: Document --with-selinux/-Dselinux options centrally > > > > This appears to have broken building the INSTALL file: > > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=guaibasaurus=2023-11-17%2020%3A20%3A03 > > > We have a mechanism for supplying alternate text for INSTALL, IIRC. > > Gah, will look at it. Pushed the fix. Greetings, Andres Freund
pgsql: docs: Fix standalone INSTALL, broken in 06c70849fb2
docs: Fix standalone INSTALL, broken in 06c70849fb2 We should probably check that INSTALL can be generated in CI. Reported-by: Tom Lane Discussion: https://postgr.es/m/795075.1700254...@sss.pgh.pa.us Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a268a51de692c462e641a58ac842bbb0ac1e43ea Modified Files -- doc/src/sgml/standalone-profile.xsl | 4 1 file changed, 4 insertions(+)
Re: pgsql: docs: Document --with-selinux/-Dselinux options centrally
Hi, On 2023-11-17 15:49:52 -0500, Tom Lane wrote: > Andres Freund writes: > > docs: Document --with-selinux/-Dselinux options centrally > > This appears to have broken building the INSTALL file: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=guaibasaurus=2023-11-17%2020%3A20%3A03 > We have a mechanism for supplying alternate text for INSTALL, IIRC. Gah, will look at it. Greetings, Andres Freund
pgsql: Release lock on heap buffer before vacuuming FSM
Release lock on heap buffer before vacuuming FSM When there are no indexes on a table, we vacuum each heap block after pruning it and then update the freespace map. Periodically, we also vacuum the freespace map. This was done while unnecessarily holding a lock on the heap page. Release the lock before calling FreeSpaceMapVacuumRange() and, while we're at it, ensure the range includes the heap block we just vacuumed. There are no known deadlocks or other similar issues, therefore don't backpatch. It's certainly not good to do all this work under a lock, but it's not frequently reached, making it not worth the risk of backpatching. Author: Melanie Plageman Reviewed-by: Andres Freund Discussion: https://postgr.es/m/CAAKRu_YiL%3D44GvGnt1dpYouDSSoV7wzxVoXs8m3p311rp-TVQQ%40mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/b2e237afddc56a4826121169a693412d8683d940 Modified Files -- src/backend/access/heap/vacuumlazy.c | 30 ++ 1 file changed, 18 insertions(+), 12 deletions(-)
pgsql: docs: Document --with-selinux/-Dselinux options centrally
docs: Document --with-selinux/-Dselinux options centrally Previously --with-selinux was documented only in the in the sepgsql documentation and there was no corresponding documentation for meson. There are further improvements that could be made, but this change seems worthwhile even on its own. Reviewed-by: Peter Eisentraut Reported-by: Christoph Berg Discussion: https://postgr.es/m/20231103163848.26egkh5qdgw3v...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/06c70849fb26ac431a722b1d10cffe1c65e728a4 Modified Files -- doc/src/sgml/installation.sgml | 21 + doc/src/sgml/sepgsql.sgml | 11 --- 2 files changed, 29 insertions(+), 3 deletions(-)
pgsql: meson: Change default of 'selinux' feature option to auto
meson: Change default of 'selinux' feature option to auto There is really no reason for selinux to behave differently than other options. Reviewed-by: Peter Eisentraut Discussion: https://postgr.es/m/20231103211601.bgqx3cfq6pz2l...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/c04f3de3dca3b54e16d7774885c8b64f90ca259b Modified Files -- meson_options.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: meson: docs: Install all manpages, not just ones in man1
meson: docs: Install all manpages, not just ones in man1 In f13eb16485f I made a mistake leading to only man1 being installed. I will report a bug suggesting that meson warn about mistakes of this sort. Reported-by: Christoph Berg Discussion: https://postgr.es/m/zuu5prqo6zueb...@msg.df7cb.de Backpatch: 16-, where the meson build was introduced Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a237a07d5351d7bb0afc6e0c41410d52915e47c8 Modified Files -- doc/src/sgml/meson.build | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
pgsql: meson: docs: Install all manpages, not just ones in man1
meson: docs: Install all manpages, not just ones in man1 In f13eb16485f I made a mistake leading to only man1 being installed. I will report a bug suggesting that meson warn about mistakes of this sort. Reported-by: Christoph Berg Discussion: https://postgr.es/m/zuu5prqo6zueb...@msg.df7cb.de Backpatch: 16-, where the meson build was introduced Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/f646ac6725672896d424e0ac46b3ba77a4063264 Modified Files -- doc/src/sgml/meson.build | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
pgsql: meson: Make detection of python more robust
meson: Make detection of python more robust Previously we errored out if no python installation could be found (but we did handle not having enough of python installed to build plpython against). Presumably nobody hit this so far, as python is likely installed due to meson requiring python. Author: Tristan Partin Discussion: https://postgr.es/m/CSPIJVUDZFKX.3KHMOAVGF94RV@c3po Backpatch: 16-, where meson support was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/41da94fd5c7418838ec7c49456b69c462013ea57 Modified Files -- meson.build | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-)
pgsql: meson: Make detection of python more robust
meson: Make detection of python more robust Previously we errored out if no python installation could be found (but we did handle not having enough of python installed to build plpython against). Presumably nobody hit this so far, as python is likely installed due to meson requiring python. Author: Tristan Partin Discussion: https://postgr.es/m/CSPIJVUDZFKX.3KHMOAVGF94RV@c3po Backpatch: 16-, where meson support was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/c79ca0485a62b2abefabd538ac896cd64f4e7a01 Modified Files -- meson.build | 12 +++- 1 file changed, 7 insertions(+), 5 deletions(-)
pgsql: Fix bulk table extension when copying into multiple partitions
Fix bulk table extension when copying into multiple partitions When COPYing into a partitioned table that does now permit the use of table_multi_insert(), we could error out with ERROR: could not read block NN in file "base/...": read only 0 of 8192 bytes because BulkInsertState->next_free was not reset between partitions. This problem occurred only when not able to use table_multi_insert(), as a dedicated BulkInsertState for each partition is used in that case. The bug was introduced in 00d1e02be24, but it was hard to hit at that point, as commonly bulk relation extension is not used when not using table_multi_insert(). It became more likely after 82a4edabd27, which expanded the use of bulk extension. To fix the bug, reset the bulk relation extension state in BulkInsertState in ReleaseBulkInsertStatePin(). That was added (in b1ecb9b3fcf) to tackle a very similar issue. Obviously the name is not quite correct, but there might be external callers, and bulk insert state needs to be reset in precisely in the situations that ReleaseBulkInsertStatePin() already needed to be called. Medium term the better fix likely is to disallow reusing BulkInsertState across relations. Add a test that, without the fix, reproduces #18130 in most configurations. The test also catches the problem fixed in b1ecb9b3fcf when run with small shared_buffers. Reported-by: Ivan Kolombet Analyzed-by: Tom Lane Analyzed-by: Andres Freund Bug: #18130 Discussion: https://postgr.es/m/18130-7a86a7356a75209d%40postgresql.org Discussion: https://postgr.es/m/257696.1695670946%40sss.pgh.pa.us Backpatch: 16- Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/0002feb8209618e5a9e23e03fe4aa31bc4006f01 Modified Files -- src/backend/access/heap/heapam.c | 11 +++ src/test/regress/expected/copy.out | 37 + src/test/regress/sql/copy.sql | 37 + 3 files changed, 85 insertions(+)
pgsql: Fix bulk table extension when copying into multiple partitions
Fix bulk table extension when copying into multiple partitions When COPYing into a partitioned table that does now permit the use of table_multi_insert(), we could error out with ERROR: could not read block NN in file "base/...": read only 0 of 8192 bytes because BulkInsertState->next_free was not reset between partitions. This problem occurred only when not able to use table_multi_insert(), as a dedicated BulkInsertState for each partition is used in that case. The bug was introduced in 00d1e02be24, but it was hard to hit at that point, as commonly bulk relation extension is not used when not using table_multi_insert(). It became more likely after 82a4edabd27, which expanded the use of bulk extension. To fix the bug, reset the bulk relation extension state in BulkInsertState in ReleaseBulkInsertStatePin(). That was added (in b1ecb9b3fcf) to tackle a very similar issue. Obviously the name is not quite correct, but there might be external callers, and bulk insert state needs to be reset in precisely in the situations that ReleaseBulkInsertStatePin() already needed to be called. Medium term the better fix likely is to disallow reusing BulkInsertState across relations. Add a test that, without the fix, reproduces #18130 in most configurations. The test also catches the problem fixed in b1ecb9b3fcf when run with small shared_buffers. Reported-by: Ivan Kolombet Analyzed-by: Tom Lane Analyzed-by: Andres Freund Bug: #18130 Discussion: https://postgr.es/m/18130-7a86a7356a75209d%40postgresql.org Discussion: https://postgr.es/m/257696.1695670946%40sss.pgh.pa.us Backpatch: 16- Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/22655aa23132a0645fdcdce4b233a1fff0c0cf8f Modified Files -- src/backend/access/heap/heapam.c | 11 +++ src/test/regress/expected/copy.out | 37 + src/test/regress/sql/copy.sql | 37 + 3 files changed, 85 insertions(+)
pgsql: meson: macos: Correct -exported_symbols_list syntax for Sonoma c
meson: macos: Correct -exported_symbols_list syntax for Sonoma compat -exported_symbols_list=... works on Ventura and earlier, but not on Sonoma. The easiest way to fix it is to -Wl,-exported_symbols_list,@0@ which actually seems more appropriate anyway, it's obviously a linker argument. It is easier to use the -Wl,, syntax than passing multiple arguments, due to the way the export_fmt is used (a single string that's formatted), but if it turns out to be necessary, we can go for multiple arguments as well. Reviewed-by: Tom Lane Discussion: https://postgr.es/m/2023092848.jw6s7yktpfsfc...@alap3.anarazel.de Backpatch: 16-, where the meson based buildsystem was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/b1a8dc846da4d96d903dcb5733f68a1e02d82a23 Modified Files -- meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: meson: macos: Correct -exported_symbols_list syntax for Sonoma c
meson: macos: Correct -exported_symbols_list syntax for Sonoma compat -exported_symbols_list=... works on Ventura and earlier, but not on Sonoma. The easiest way to fix it is to -Wl,-exported_symbols_list,@0@ which actually seems more appropriate anyway, it's obviously a linker argument. It is easier to use the -Wl,, syntax than passing multiple arguments, due to the way the export_fmt is used (a single string that's formatted), but if it turns out to be necessary, we can go for multiple arguments as well. Reviewed-by: Tom Lane Discussion: https://postgr.es/m/2023092848.jw6s7yktpfsfc...@alap3.anarazel.de Backpatch: 16-, where the meson based buildsystem was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/d5c5312dc8b66b4ad652ec54ed32029d621e1e93 Modified Files -- meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: pg_dump: tests: Correct test condition for invalid databases
pg_dump: tests: Correct test condition for invalid databases
For some reason I used not_like = { pg_dumpall_dbprivs => 1, } in the test
condition of one of the tests added in in c66a7d75e65. That doesn't make sense
for two reasons: 1) not_like isn't a valid test condition 2) the database
should not be dumped in any of the tests. Due to 1), the test achieved its
goal, but clearly the formulation is confusing. Instead use like => {}, with
a comment explaining why.
Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/3ddf79f2-8b7b-a093-11d2-5c739bc64...@eisentraut.org
Backpatch: 11-, like c66a7d75e65
Branch
--
REL_11_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/6a3b19bbefd782605765c67990e9d8b4dd5e9f87
Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
pgsql: pg_dump: tests: Correct test condition for invalid databases
pg_dump: tests: Correct test condition for invalid databases
For some reason I used not_like = { pg_dumpall_dbprivs => 1, } in the test
condition of one of the tests added in in c66a7d75e65. That doesn't make sense
for two reasons: 1) not_like isn't a valid test condition 2) the database
should not be dumped in any of the tests. Due to 1), the test achieved its
goal, but clearly the formulation is confusing. Instead use like => {}, with
a comment explaining why.
Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/3ddf79f2-8b7b-a093-11d2-5c739bc64...@eisentraut.org
Backpatch: 11-, like c66a7d75e65
Branch
--
REL_15_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/9dc3c5472ed99a1046a4d7884270a35cfad29d1d
Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
pgsql: pg_dump: tests: Correct test condition for invalid databases
pg_dump: tests: Correct test condition for invalid databases
For some reason I used not_like = { pg_dumpall_dbprivs => 1, } in the test
condition of one of the tests added in in c66a7d75e65. That doesn't make sense
for two reasons: 1) not_like isn't a valid test condition 2) the database
should not be dumped in any of the tests. Due to 1), the test achieved its
goal, but clearly the formulation is confusing. Instead use like => {}, with
a comment explaining why.
Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/3ddf79f2-8b7b-a093-11d2-5c739bc64...@eisentraut.org
Backpatch: 11-, like c66a7d75e65
Branch
--
REL_14_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/4dfb610822d5ab4bea7820da9be02d83bad69fb5
Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
pgsql: pg_dump: tests: Correct test condition for invalid databases
pg_dump: tests: Correct test condition for invalid databases
For some reason I used not_like = { pg_dumpall_dbprivs => 1, } in the test
condition of one of the tests added in in c66a7d75e65. That doesn't make sense
for two reasons: 1) not_like isn't a valid test condition 2) the database
should not be dumped in any of the tests. Due to 1), the test achieved its
goal, but clearly the formulation is confusing. Instead use like => {}, with
a comment explaining why.
Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/3ddf79f2-8b7b-a093-11d2-5c739bc64...@eisentraut.org
Backpatch: 11-, like c66a7d75e65
Branch
--
master
Details
---
https://git.postgresql.org/pg/commitdiff/849d367ff9a2875d4906fa110472462c4c95fad0
Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
pgsql: pg_dump: tests: Correct test condition for invalid databases
pg_dump: tests: Correct test condition for invalid databases
For some reason I used not_like = { pg_dumpall_dbprivs => 1, } in the test
condition of one of the tests added in in c66a7d75e65. That doesn't make sense
for two reasons: 1) not_like isn't a valid test condition 2) the database
should not be dumped in any of the tests. Due to 1), the test achieved its
goal, but clearly the formulation is confusing. Instead use like => {}, with
a comment explaining why.
Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/3ddf79f2-8b7b-a093-11d2-5c739bc64...@eisentraut.org
Backpatch: 11-, like c66a7d75e65
Branch
--
REL_12_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/fd7a114dbb4e8b700b0b738bb9c513b5fa3198a3
Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
pgsql: pg_dump: tests: Correct test condition for invalid databases
pg_dump: tests: Correct test condition for invalid databases
For some reason I used not_like = { pg_dumpall_dbprivs => 1, } in the test
condition of one of the tests added in in c66a7d75e65. That doesn't make sense
for two reasons: 1) not_like isn't a valid test condition 2) the database
should not be dumped in any of the tests. Due to 1), the test achieved its
goal, but clearly the formulation is confusing. Instead use like => {}, with
a comment explaining why.
Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/3ddf79f2-8b7b-a093-11d2-5c739bc64...@eisentraut.org
Backpatch: 11-, like c66a7d75e65
Branch
--
REL_13_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/d1c76fdec3751c724a25d449ea5aeeb33a27b3d3
Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
pgsql: pg_dump: tests: Correct test condition for invalid databases
pg_dump: tests: Correct test condition for invalid databases
For some reason I used not_like = { pg_dumpall_dbprivs => 1, } in the test
condition of one of the tests added in in c66a7d75e65. That doesn't make sense
for two reasons: 1) not_like isn't a valid test condition 2) the database
should not be dumped in any of the tests. Due to 1), the test achieved its
goal, but clearly the formulation is confusing. Instead use like => {}, with
a comment explaining why.
Reported-by: Peter Eisentraut
Discussion:
https://postgr.es/m/3ddf79f2-8b7b-a093-11d2-5c739bc64...@eisentraut.org
Backpatch: 11-, like c66a7d75e65
Branch
--
REL_16_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/881defde944c07d821d9658b1943d5ad5ef5bf37
Modified Files
--
src/bin/pg_dump/t/002_pg_dump.pl | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
pgsql: docs: Clarify --with-segsize-blocks documentation
docs: Clarify --with-segsize-blocks documentation Without the added "relation" it's not immediately clear that the option relates to the relation segment size and not e.g. the WAL segment size. The option was added in d3b111e32. Reported-by: Tom Lane Discussion: https://postgr.es/m/837536.1695348...@sss.pgh.pa.us Backpatch: 16- Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/72bf3f0877fe7ae9a22c23f461d0fdafb6af5c4b Modified Files -- doc/src/sgml/installation.sgml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
pgsql: docs: Clarify --with-segsize-blocks documentation
docs: Clarify --with-segsize-blocks documentation Without the added "relation" it's not immediately clear that the option relates to the relation segment size and not e.g. the WAL segment size. The option was added in d3b111e32. Reported-by: Tom Lane Discussion: https://postgr.es/m/837536.1695348...@sss.pgh.pa.us Backpatch: 16- Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a2c2fbf740fcfdf75b76d589b8e85c1cdf725a46 Modified Files -- doc/src/sgml/installation.sgml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
pgsql: Fix tracking of temp table relation extensions as writes
Fix tracking of temp table relation extensions as writes Karina figured out that I (Andres) confused BufferUsage.temp_blks_written with BufferUsage.local_blks_written in fcdda1e4b5. Tests in core PG can't easily test this, as BufferUsage is just used for EXPLAIN (ANALYZE, BUFFERS) and pg_stat_statements. Thus this commit adds tests for this to pg_stat_statements. Reported-by: Karina Litskevich Author: Karina Litskevich Author: Andres Freund Discussion: https://postgr.es/m/cacit8ibxxa6+0amgikbefhm8b84xdqvo6d0qfd1pq1s8zps...@mail.gmail.com Backpatch: 16-, where fcdda1e4b5 was merged Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/c4758649b4db7d0c4b681985d24cc3e1e938047d Modified Files -- contrib/pg_stat_statements/expected/dml.out | 27 +++ contrib/pg_stat_statements/sql/dml.sql | 19 +++ src/backend/storage/buffer/localbuf.c | 2 +- 3 files changed, 47 insertions(+), 1 deletion(-)
pgsql: Fix tracking of temp table relation extensions as writes
Fix tracking of temp table relation extensions as writes Karina figured out that I (Andres) confused BufferUsage.temp_blks_written with BufferUsage.local_blks_written in fcdda1e4b5. Tests in core PG can't easily test this, as BufferUsage is just used for EXPLAIN (ANALYZE, BUFFERS) and pg_stat_statements. Thus this commit adds tests for this to pg_stat_statements. Reported-by: Karina Litskevich Author: Karina Litskevich Author: Andres Freund Discussion: https://postgr.es/m/cacit8ibxxa6+0amgikbefhm8b84xdqvo6d0qfd1pq1s8zps...@mail.gmail.com Backpatch: 16-, where fcdda1e4b5 was merged Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/7369798a83c883a6314b8a8ed6d3063c508de784 Modified Files -- contrib/pg_stat_statements/expected/dml.out | 27 +++ contrib/pg_stat_statements/sql/dml.sql | 19 +++ src/backend/storage/buffer/localbuf.c | 2 +- 3 files changed, 47 insertions(+), 1 deletion(-)
pgsql: Avoid non-POSIX cp flags
Avoid non-POSIX cp flags Commit 252dcb32 used cp -a, but apparently Solaris doesn't like that. Use cp -RPp instead. Author: Thomas Munro Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/ca+hukgl10aoqvmmqgoj8ctjoz9mlidd8ik2e8pibzlnmz0+...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/1a4fd77db85abac63e178506335aee74625f6499 Modified Files -- src/test/perl/PostgreSQL/Test/Cluster.pm | 2 +- src/test/regress/pg_regress.c| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
pgsql: Use "template" data directory in tests
Use "template" data directory in tests When running all (or just many) of our tests, a significant portion of both CPU time and IO is spent running initdb. Most of those initdb runs don't specify any options influencing properties of the created data directory. Avoid most of that overhead by creating a "template" data directory, alongside the temporary installation. Instead of running initdb, pg_regress and tap tests can copy that data directory. When a tap test specifies options to initdb, the template data directory is not used. That could be relaxed for some options, but it's not clear it's worth the effort. There unfortunately is some duplication between pg_regress.c and Cluster.pm, but there are no easy ways of sharing that code without introducing additional complexity. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20220120021859.3zpsfqn4z7ob7...@alap3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/252dcb32397f64a5e1ceac05b29a271ab19aa960 Modified Files -- .cirrus.tasks.yml| 3 +- meson.build | 30 + src/Makefile.global.in | 38 +--- src/test/perl/PostgreSQL/Test/Cluster.pm | 46 ++- src/test/regress/pg_regress.c| 76 +--- 5 files changed, 156 insertions(+), 37 deletions(-)
pgsql: ci: Make compute resources for CI configurable
ci: Make compute resources for CI configurable See prior commit for an explanation for the goal of the change and why it had to be split into two commits. Reviewed-by: Daniel Gustafsson Reviewed-by: Nazir Bilal Yavuz Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/e8a8cd05d4b9533cf5bff0f4a4ee83a40df62875 Modified Files -- .cirrus.tasks.yml | 45 - .cirrus.yml | 73 + src/tools/ci/README | 17 + 3 files changed, 90 insertions(+), 45 deletions(-)
pgsql: ci: Prepare to make compute resources for CI configurable
ci: Prepare to make compute resources for CI configurable cirrus-ci will soon restrict the amount of free resources every user gets (as have many other CI providers). For most users of CI that should not be an issue. But e.g. for cfbot it will be an issue. To allow configuring different resources on a per-repository basis, introduce infrastructure for overriding the task execution environment. Unfortunately this is not entirely trivial, as yaml anchors have to be defined before their use, and cirrus-ci only allows injecting additional contents at the end of .cirrus.yml. To deal with that, move the definition of the CI tasks to .cirrus.tasks.yml. The main .cirrus.yml is loaded first, then, if defined, the file referenced by the REPO_CI_CONFIG_GIT_URL variable, will be added, followed by the contents of .cirrus.tasks.yml. That allows REPO_CI_CONFIG_GIT_URL to override the yaml anchors defined in .cirrus.yml. Unfortunately git's default merge / rebase strategy does not handle copied files, just renamed ones. To avoid painful rebasing over this change, this commit just renames .cirrus.yml to .cirrus.tasks.yml, without adding a new .cirrus.yml. That's done in the followup commit, which moves the relevant portion of .cirrus.tasks.yml to .cirrus.yml. Until that is done, REPO_CI_CONFIG_GIT_URL does not fully work. The subsequent commit adds documentation for how to configure custom compute resources to src/tools/ci/README Reviewed-by: Daniel Gustafsson Reviewed-by: Nazir Bilal Yavuz Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/19cc96503d23c79900162396077a900e7507c4d3 Modified Files -- .cirrus.star | 63 .cirrus.yml => .cirrus.tasks.yml | 0 2 files changed, 63 insertions(+)
pgsql: ci: Prepare to make compute resources for CI configurable
ci: Prepare to make compute resources for CI configurable cirrus-ci will soon restrict the amount of free resources every user gets (as have many other CI providers). For most users of CI that should not be an issue. But e.g. for cfbot it will be an issue. To allow configuring different resources on a per-repository basis, introduce infrastructure for overriding the task execution environment. Unfortunately this is not entirely trivial, as yaml anchors have to be defined before their use, and cirrus-ci only allows injecting additional contents at the end of .cirrus.yml. To deal with that, move the definition of the CI tasks to .cirrus.tasks.yml. The main .cirrus.yml is loaded first, then, if defined, the file referenced by the REPO_CI_CONFIG_GIT_URL variable, will be added, followed by the contents of .cirrus.tasks.yml. That allows REPO_CI_CONFIG_GIT_URL to override the yaml anchors defined in .cirrus.yml. Unfortunately git's default merge / rebase strategy does not handle copied files, just renamed ones. To avoid painful rebasing over this change, this commit just renames .cirrus.yml to .cirrus.tasks.yml, without adding a new .cirrus.yml. That's done in the followup commit, which moves the relevant portion of .cirrus.tasks.yml to .cirrus.yml. Until that is done, REPO_CI_CONFIG_GIT_URL does not fully work. The subsequent commit adds documentation for how to configure custom compute resources to src/tools/ci/README Reviewed-by: Daniel Gustafsson Reviewed-by: Nazir Bilal Yavuz Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/9ed46c78a3625001a863bbcbfd5f1c696c1def46 Modified Files -- .cirrus.star | 63 .cirrus.yml => .cirrus.tasks.yml | 0 2 files changed, 63 insertions(+)
pgsql: ci: Make compute resources for CI configurable
ci: Make compute resources for CI configurable See prior commit for an explanation for the goal of the change and why it had to be split into two commits. Reviewed-by: Daniel Gustafsson Reviewed-by: Nazir Bilal Yavuz Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/e4693c68a497129ea3913148027a6cee3b522d5d Modified Files -- .cirrus.tasks.yml | 45 - .cirrus.yml | 73 + src/tools/ci/README | 17 + 3 files changed, 90 insertions(+), 45 deletions(-)
pgsql: ci: Prepare to make compute resources for CI configurable
ci: Prepare to make compute resources for CI configurable cirrus-ci will soon restrict the amount of free resources every user gets (as have many other CI providers). For most users of CI that should not be an issue. But e.g. for cfbot it will be an issue. To allow configuring different resources on a per-repository basis, introduce infrastructure for overriding the task execution environment. Unfortunately this is not entirely trivial, as yaml anchors have to be defined before their use, and cirrus-ci only allows injecting additional contents at the end of .cirrus.yml. To deal with that, move the definition of the CI tasks to .cirrus.tasks.yml. The main .cirrus.yml is loaded first, then, if defined, the file referenced by the REPO_CI_CONFIG_GIT_URL variable, will be added, followed by the contents of .cirrus.tasks.yml. That allows REPO_CI_CONFIG_GIT_URL to override the yaml anchors defined in .cirrus.yml. Unfortunately git's default merge / rebase strategy does not handle copied files, just renamed ones. To avoid painful rebasing over this change, this commit just renames .cirrus.yml to .cirrus.tasks.yml, without adding a new .cirrus.yml. That's done in the followup commit, which moves the relevant portion of .cirrus.tasks.yml to .cirrus.yml. Until that is done, REPO_CI_CONFIG_GIT_URL does not fully work. The subsequent commit adds documentation for how to configure custom compute resources to src/tools/ci/README Reviewed-by: Daniel Gustafsson Reviewed-by: Nazir Bilal Yavuz Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/284465e1b95ef5f561b0fadead2468c0c63c0f82 Modified Files -- .cirrus.star | 63 .cirrus.yml => .cirrus.tasks.yml | 0 2 files changed, 63 insertions(+)
pgsql: ci: Make compute resources for CI configurable
ci: Make compute resources for CI configurable See prior commit for an explanation for the goal of the change and why it had to be split into two commits. Reviewed-by: Daniel Gustafsson Reviewed-by: Nazir Bilal Yavuz Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a28166df8c5a00ec3d9b3550a4f6598593ebf873 Modified Files -- .cirrus.tasks.yml | 45 - .cirrus.yml | 73 + src/tools/ci/README | 17 + 3 files changed, 90 insertions(+), 45 deletions(-)
pgsql: ci: Move execution method of tasks into yaml templates
ci: Move execution method of tasks into yaml templates This is done in preparation for making the compute resources for CI configurable. It also looks cleaner. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/cad461b044b521c5f80d180194d6e27dce82c424 Modified Files -- .cirrus.yml | 85 + 1 file changed, 57 insertions(+), 28 deletions(-)
pgsql: ci: Don't specify amount of memory
ci: Don't specify amount of memory The number of CPUs is the cost-determining factor. Most instance types that run tests have more memory/core than what we specified, there's no real benefit in wasting that. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/5581a9a3953020dd9df18ffd00b788f296bd60e0 Modified Files -- .cirrus.yml | 4 1 file changed, 4 deletions(-)
pgsql: ci: Don't specify amount of memory
ci: Don't specify amount of memory The number of CPUs is the cost-determining factor. Most instance types that run tests have more memory/core than what we specified, there's no real benefit in wasting that. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/89daa5ae307a56017ce47e7dc652284ce547012a Modified Files -- .cirrus.yml | 3 --- 1 file changed, 3 deletions(-)
pgsql: ci: Use VMs for SanityCheck and CompilerWarnings
ci: Use VMs for SanityCheck and CompilerWarnings The main reason for this change is to reduce different ways of executing tasks, making it easier to use custom compute resources for cfbot. A secondary benefit is that the tasks seem slightly faster this way, apparently the increased startup overhead is outweighed by reduced runtime overhead. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/4fdfd0629d77af19582ca4d59c0e7fe99619f048 Modified Files -- .cirrus.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
pgsql: ci: Don't specify amount of memory
ci: Don't specify amount of memory The number of CPUs is the cost-determining factor. Most instance types that run tests have more memory/core than what we specified, there's no real benefit in wasting that. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/794e14e219c7997c7f59ea8b7923daee95d2a3a6 Modified Files -- .cirrus.yml | 4 1 file changed, 4 deletions(-)
pgsql: ci: Use VMs for SanityCheck and CompilerWarnings
ci: Use VMs for SanityCheck and CompilerWarnings The main reason for this change is to reduce different ways of executing tasks, making it easier to use custom compute resources for cfbot. A secondary benefit is that the tasks seem slightly faster this way, apparently the increased startup overhead is outweighed by reduced runtime overhead. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/b2c91d841f1f58f6f553d9e6832fdfaedec00402 Modified Files -- .cirrus.yml | 14 ++ 1 file changed, 6 insertions(+), 8 deletions(-)
pgsql: ci: Move execution method of tasks into yaml templates
ci: Move execution method of tasks into yaml templates This is done in preparation for making the compute resources for CI configurable. It also looks cleaner. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/462f4df0a86d2c497a1739f386835d246ea8645e Modified Files -- .cirrus.yml | 77 +++-- 1 file changed, 55 insertions(+), 22 deletions(-)
pgsql: ci: Use VMs for SanityCheck and CompilerWarnings
ci: Use VMs for SanityCheck and CompilerWarnings The main reason for this change is to reduce different ways of executing tasks, making it easier to use custom compute resources for cfbot. A secondary benefit is that the tasks seem slightly faster this way, apparently the increased startup overhead is outweighed by reduced runtime overhead. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/f518c909ead1be7c88ee1f53db13bdef7c50d1ca Modified Files -- .cirrus.yml | 14 ++ 1 file changed, 6 insertions(+), 8 deletions(-)
pgsql: ci: Move execution method of tasks into yaml templates
ci: Move execution method of tasks into yaml templates This is done in preparation for making the compute resources for CI configurable. It also looks cleaner. Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 15-, where CI support was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/119ee6ab1b0081e339cac08d18ae625fcc77bcd1 Modified Files -- .cirrus.yml | 85 + 1 file changed, 57 insertions(+), 28 deletions(-)
pgsql: ci: macos: use cached macports install
ci: macos: use cached macports install A significant chunk of the time on the macos CI task is spent installing packages using homebrew. The downloads of the packages are cached, but the installation needs to happen every time. We can't cache the whole homebrew installation, because it is too large due to pre-installed packages. Speed this up by installing packages using macports and caching the installation as .dmg. That's a lot faster than unpacking a tarball. In addition, don't install llvm - it wasn't enabled when building, so it's just a waste of time/space. This substantially speeds up the mac CI time, both in the cold cache and in the warm cache case (the latter from ~1m20s to ~5s). It doesn't seem great to have diverging sources of packages for CI between branches, so backpatch to 15 (where CI was added). Discussion: https://postgr.es/m/20230805202539.r3umyamsnctys...@awork3.anarazel.de Backpatch: 15-, where CI was added Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/62017cb51ff9f1ff6aa94bdf26d9cc290d4ffe48 Modified Files -- .cirrus.yml | 67 ++--- src/tools/ci/ci_macports_packages.sh | 97 2 files changed, 124 insertions(+), 40 deletions(-)
pgsql: ci: macos: use cached macports install
ci: macos: use cached macports install A significant chunk of the time on the macos CI task is spent installing packages using homebrew. The downloads of the packages are cached, but the installation needs to happen every time. We can't cache the whole homebrew installation, because it is too large due to pre-installed packages. Speed this up by installing packages using macports and caching the installation as .dmg. That's a lot faster than unpacking a tarball. In addition, don't install llvm - it wasn't enabled when building, so it's just a waste of time/space. This substantially speeds up the mac CI time, both in the cold cache and in the warm cache case (the latter from ~1m20s to ~5s). It doesn't seem great to have diverging sources of packages for CI between branches, so backpatch to 15 (where CI was added). Discussion: https://postgr.es/m/20230805202539.r3umyamsnctys...@awork3.anarazel.de Backpatch: 15-, where CI was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/a2a6249cf1a4210caac534e8454a1614d0dd081a Modified Files -- .cirrus.yml | 63 ++- src/tools/ci/ci_macports_packages.sh | 97 2 files changed, 122 insertions(+), 38 deletions(-)
pgsql: ci: macos: use cached macports install
ci: macos: use cached macports install A significant chunk of the time on the macos CI task is spent installing packages using homebrew. The downloads of the packages are cached, but the installation needs to happen every time. We can't cache the whole homebrew installation, because it is too large due to pre-installed packages. Speed this up by installing packages using macports and caching the installation as .dmg. That's a lot faster than unpacking a tarball. In addition, don't install llvm - it wasn't enabled when building, so it's just a waste of time/space. This substantially speeds up the mac CI time, both in the cold cache and in the warm cache case (the latter from ~1m20s to ~5s). It doesn't seem great to have diverging sources of packages for CI between branches, so backpatch to 15 (where CI was added). Discussion: https://postgr.es/m/20230805202539.r3umyamsnctys...@awork3.anarazel.de Backpatch: 15-, where CI was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/dd2a731d9caffa8c9f6d82f2b72dd460b67decd7 Modified Files -- .cirrus.yml | 63 ++- src/tools/ci/ci_macports_packages.sh | 97 2 files changed, 122 insertions(+), 38 deletions(-)
pgsql: hio: Take number of prior relation extensions into account
hio: Take number of prior relation extensions into account The new relation extension logic, introduced in 00d1e02be24, could lead to slowdowns in some scenarios. E.g., when loading narrow rows into a table using COPY, the caller of RelationGetBufferForTuple() will only request a small number of pages. Without concurrency, we just extended using pwritev() in that case. However, if there is *some* concurrency, we switched between extending by a small number of pages and a larger number of pages, depending on the number of waiters for the relation extension logic. However, some filesystems, XFS in particular, do not perform well when switching between extending files using fallocate() and pwritev(). To avoid that issue, remember the number of prior relation extensions in BulkInsertState and extend more aggressively if there were prior relation extensions. That not just avoids the aforementioned slowdown, but also leads to noticeable performance gains in other situations, primarily due to extending more aggressively when there is no concurrency. I should have done it this way from the get go. Reported-by: Masahiko Sawada Author: Andres Freund Discussion: https://postgr.es/m/CAD21AoDvDmUQeJtZrau1ovnT_smN940=kp6mszngk3bq9yr...@mail.gmail.com Backpatch: 16-, where the new relation extension code was added Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/82a4edabd272f70d044faec8cf7fd1eab92d9991 Modified Files -- src/backend/access/heap/heapam.c | 1 + src/backend/access/heap/hio.c| 19 +++ src/include/access/hio.h | 13 ++--- 3 files changed, 30 insertions(+), 3 deletions(-)
pgsql: hio: Take number of prior relation extensions into account
hio: Take number of prior relation extensions into account The new relation extension logic, introduced in 00d1e02be24, could lead to slowdowns in some scenarios. E.g., when loading narrow rows into a table using COPY, the caller of RelationGetBufferForTuple() will only request a small number of pages. Without concurrency, we just extended using pwritev() in that case. However, if there is *some* concurrency, we switched between extending by a small number of pages and a larger number of pages, depending on the number of waiters for the relation extension logic. However, some filesystems, XFS in particular, do not perform well when switching between extending files using fallocate() and pwritev(). To avoid that issue, remember the number of prior relation extensions in BulkInsertState and extend more aggressively if there were prior relation extensions. That not just avoids the aforementioned slowdown, but also leads to noticeable performance gains in other situations, primarily due to extending more aggressively when there is no concurrency. I should have done it this way from the get go. Reported-by: Masahiko Sawada Author: Andres Freund Discussion: https://postgr.es/m/CAD21AoDvDmUQeJtZrau1ovnT_smN940=kp6mszngk3bq9yr...@mail.gmail.com Backpatch: 16-, where the new relation extension code was added Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/d37ab378b6e773c278c14b9554a1ea23b355aab9 Modified Files -- src/backend/access/heap/heapam.c | 1 + src/backend/access/heap/hio.c| 19 +++ src/include/access/hio.h | 13 ++--- 3 files changed, 30 insertions(+), 3 deletions(-)
pgsql: ci: macos: Remove use of -Dsegsize_blocks=6
ci: macos: Remove use of -Dsegsize_blocks=6 The option causes a measurable slowdown. Macos is, by far, the most expensive platform for CI, therefore it doesn't make sense to run such a test there. d3b111e3205 used a small segment size for two tasks, one with autoconf, one with meson. In hindsight that is a bit overkill, it's unlikely that the option would silently break. Thus don't move the -Dsegsize_blocks=6, just remove it. I did however change the autoconf test to use 6 instead of 8 blocks, as long as we allow it, a non-power-of-two test seems like a good idea. While at it, add a comment explaining why we use a small segment size for CI. Author: Andres Freund Reviewed-by: Heikki Linnakangas Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 16-, where d3b111e3205 introduced the use of -Dsegsize_blocks=6 Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/3d8d217450a63638825167c17ed791122f376176 Modified Files -- .cirrus.yml | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-)
pgsql: ci: macos: Remove use of -Dsegsize_blocks=6
ci: macos: Remove use of -Dsegsize_blocks=6 The option causes a measurable slowdown. Macos is, by far, the most expensive platform for CI, therefore it doesn't make sense to run such a test there. d3b111e3205 used a small segment size for two tasks, one with autoconf, one with meson. In hindsight that is a bit overkill, it's unlikely that the option would silently break. Thus don't move the -Dsegsize_blocks=6, just remove it. I did however change the autoconf test to use 6 instead of 8 blocks, as long as we allow it, a non-power-of-two test seems like a good idea. While at it, add a comment explaining why we use a small segment size for CI. Author: Andres Freund Reviewed-by: Heikki Linnakangas Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 16-, where d3b111e3205 introduced the use of -Dsegsize_blocks=6 Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/2243ef8dd6fbe1f4ae3ffc7db1ee9af9427ecadb Modified Files -- .cirrus.yml | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-)
pgsql: ci: macos: Remove use of -DRANDOMIZE_ALLOCATED_MEMORY
ci: macos: Remove use of -DRANDOMIZE_ALLOCATED_MEMORY RANDOMIZE_ALLOCATED_MEMORY causes a measurable slowdown. Macos is, by far, the most expensive platform for CI, therefore it doesn't make sense to run such a test there. Ubsan and asan on linux should detect most of the the cases of uninitialized memory, so it doesn't really seem worth using -DRANDOMIZE_ALLOCATED_MEMORY in another instance type. Author: Andres Freund Reviewed-by: Heikki Linnakangas Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 16-, where 89d16b635 added the use of -DRANDOMIZE_ALLOCATED_MEMORY Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/4bec616f26f5f39a6819804f2b7992495a1a7a6f Modified Files -- .cirrus.yml | 1 - 1 file changed, 1 deletion(-)
pgsql: ci: macos: Remove use of -DRANDOMIZE_ALLOCATED_MEMORY
ci: macos: Remove use of -DRANDOMIZE_ALLOCATED_MEMORY RANDOMIZE_ALLOCATED_MEMORY causes a measurable slowdown. Macos is, by far, the most expensive platform for CI, therefore it doesn't make sense to run such a test there. Ubsan and asan on linux should detect most of the the cases of uninitialized memory, so it doesn't really seem worth using -DRANDOMIZE_ALLOCATED_MEMORY in another instance type. Author: Andres Freund Reviewed-by: Heikki Linnakangas Discussion: https://postgr.es/m/20230808021541.7lbzdefvma7qm...@awork3.anarazel.de Backpatch: 16-, where 89d16b635 added the use of -DRANDOMIZE_ALLOCATED_MEMORY Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/17ebbdf7de192c340762cb852ce615aefa191c95 Modified Files -- .cirrus.yml | 1 - 1 file changed, 1 deletion(-)
pgsql: Fix pg_stat_io buffer reuse test instability
Fix pg_stat_io buffer reuse test instability The stats regression test attempts to ensure that Buffer Access Strategy "reuses" are being counted in pg_stat_io by vacuuming a table which is larger than the size of the strategy ring. However, when shared buffers are in sufficiently high demand, another backend could evict one of the blocks in the strategy ring before the first backend has a chance to reuse the buffer. The backend using the strategy would then evict another shared buffer and add that buffer to the strategy ring. This counts as an eviction and not a reuse in pg_stat_io. Count both evictions and reuses in the test to ensure it does not fail incorrectly. Reported-by: Jeff Davis , Author: Melanie Plageman Reviewed-by: Alexander Lakhin Reviewed-by: Masahiko Sawada Discussion: https://postgr.es/m/CAAKRu_bNG27AxG9TdPtwsL6wg8AWbVckjmTL2t1HF=midqu...@mail.gmail.com Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/803660ea4c5d9d2b3c7e1b0a9fc4b99359ba0a85 Modified Files -- src/test/regress/expected/stats.out | 30 -- src/test/regress/sql/stats.sql | 19 --- 2 files changed, 32 insertions(+), 17 deletions(-)
pgsql: Fix pg_stat_io buffer reuse test instability
Fix pg_stat_io buffer reuse test instability The stats regression test attempts to ensure that Buffer Access Strategy "reuses" are being counted in pg_stat_io by vacuuming a table which is larger than the size of the strategy ring. However, when shared buffers are in sufficiently high demand, another backend could evict one of the blocks in the strategy ring before the first backend has a chance to reuse the buffer. The backend using the strategy would then evict another shared buffer and add that buffer to the strategy ring. This counts as an eviction and not a reuse in pg_stat_io. Count both evictions and reuses in the test to ensure it does not fail incorrectly. Reported-by: Jeff Davis , Author: Melanie Plageman Reviewed-by: Alexander Lakhin Reviewed-by: Masahiko Sawada Discussion: https://postgr.es/m/CAAKRu_bNG27AxG9TdPtwsL6wg8AWbVckjmTL2t1HF=midqu...@mail.gmail.com Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/36ab831f9abc6f6b0ab74dd62e0faad75dbf42bf Modified Files -- src/test/regress/expected/stats.out | 30 -- src/test/regress/sql/stats.sql | 19 --- 2 files changed, 32 insertions(+), 17 deletions(-)
pgsql: Fix off-by-one in LimitAdditionalPins()
Fix off-by-one in LimitAdditionalPins()
Due to the bug LimitAdditionalPins() could return 0, violating
LimitAdditionalPins()'s API ("One additional pin is always allowed"). This
could be hit when setting shared_buffers very low and using a fair amount of
concurrency.
This bug was introduced in 31966b151e6a.
Author: "Anton A. Melnikov"
Reported-by: "Anton A. Melnikov"
Reported-by: Victoria Shepard
Discussion: https://postgr.es/m/ae46f2fb-5586-3de0-b54b-1bb0f6410...@inbox.ru
Backpatch: 16-
Branch
--
REL_16_STABLE
Details
---
https://git.postgresql.org/pg/commitdiff/bd2f46c6559ddcafe7bae5015ac45b69bc014067
Modified Files
--
src/backend/storage/buffer/bufmgr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: Fix off-by-one in LimitAdditionalPins()
Fix off-by-one in LimitAdditionalPins()
Due to the bug LimitAdditionalPins() could return 0, violating
LimitAdditionalPins()'s API ("One additional pin is always allowed"). This
could be hit when setting shared_buffers very low and using a fair amount of
concurrency.
This bug was introduced in 31966b151e6a.
Author: "Anton A. Melnikov"
Reported-by: "Anton A. Melnikov"
Reported-by: Victoria Shepard
Discussion: https://postgr.es/m/ae46f2fb-5586-3de0-b54b-1bb0f6410...@inbox.ru
Backpatch: 16-
Branch
--
master
Details
---
https://git.postgresql.org/pg/commitdiff/f3bc5192889f6f02aa10ca9f24df4eab1f1493c1
Modified Files
--
src/backend/storage/buffer/bufmgr.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
pgsql: Release lock after encountering bogs row in vac_truncate_clog()
Release lock after encountering bogs row in vac_truncate_clog() When vac_truncate_clog() encounters bogus datfrozenxid / datminmxid values, it returns early. Unfortunately, until now, it did not release WrapLimitsVacuumLock. If the backend later tries to acquire WrapLimitsVacuumLock, the session / autovacuum worker hangs in an uncancellable way. Similarly, other sessions will hang waiting for the lock. However, if the backend holding the lock exited or errored out for some reason, the lock was released. The bug was introduced as a side effect of 566372b3d643. It is interesting that there are no production reports of this problem. That is likely due to a mix of bugs leading to bogus values having gotten less common, process exit releasing locks and instances of hangs being hard to debug for "normal" users. Discussion: https://postgr.es/m/20230621221208.vhsqgduwfpzwx...@awork3.anarazel.de Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/03ccc9569cd87fcbe01e4ba052c22ad0d9a3b71d Modified Files -- src/backend/commands/vacuum.c | 4 1 file changed, 4 insertions(+)
pgsql: Release lock after encountering bogs row in vac_truncate_clog()
Release lock after encountering bogs row in vac_truncate_clog() When vac_truncate_clog() encounters bogus datfrozenxid / datminmxid values, it returns early. Unfortunately, until now, it did not release WrapLimitsVacuumLock. If the backend later tries to acquire WrapLimitsVacuumLock, the session / autovacuum worker hangs in an uncancellable way. Similarly, other sessions will hang waiting for the lock. However, if the backend holding the lock exited or errored out for some reason, the lock was released. The bug was introduced as a side effect of 566372b3d643. It is interesting that there are no production reports of this problem. That is likely due to a mix of bugs leading to bogus values having gotten less common, process exit releasing locks and instances of hangs being hard to debug for "normal" users. Discussion: https://postgr.es/m/20230621221208.vhsqgduwfpzwx...@awork3.anarazel.de Branch -- REL_15_STABLE Details --- https://git.postgresql.org/pg/commitdiff/82e97b864004810db2d335cf26117f7f1c2ff435 Modified Files -- src/backend/commands/vacuum.c | 4 1 file changed, 4 insertions(+)
pgsql: Handle DROP DATABASE getting interrupted
Handle DROP DATABASE getting interrupted Until now, when DROP DATABASE got interrupted in the wrong moment, the removal of the pg_database row would also roll back, even though some irreversible steps have already been taken. E.g. DropDatabaseBuffers() might have thrown out dirty buffers, or files could have been unlinked. But we continued to allow connections to such a corrupted database. To fix this, mark databases invalid with an in-place update, just before starting to perform irreversible steps. As we can't add a new column in the back branches, we use pg_database.datconnlimit = -2 for this purpose. An invalid database cannot be connected to anymore, but can still be dropped. Unfortunately we can't easily add output to psql's \l to indicate that some database is invalid, it doesn't fit in any of the existing columns. Add tests verifying that a interrupted DROP DATABASE is handled correctly in the backend and in various tools. Reported-by: Evgeny Morozov Author: Andres Freund Reviewed-by: Daniel Gustafsson Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/20230509004637.cgvmfwrbht7xm...@awork3.anarazel.de Discussion: https://postgr.es/m/20230314174521.74jl6ffqsee5m...@awork3.anarazel.de Backpatch: 11-, bug present in all supported versions Branch -- REL_16_STABLE Details --- https://git.postgresql.org/pg/commitdiff/a4b4cc1d60f7e8ccfcc8ff8cb80c28ee411ad9a9 Modified Files -- doc/src/sgml/catalogs.sgml | 3 +- src/backend/commands/dbcommands.c | 110 ++- src/backend/commands/vacuum.c | 14 +++ src/backend/postmaster/autovacuum.c | 12 +++ src/backend/utils/init/postinit.c | 10 +++ src/bin/pg_amcheck/pg_amcheck.c | 2 +- src/bin/pg_amcheck/t/002_nonesuch.pl| 34 +++ src/bin/pg_dump/pg_dumpall.c| 4 +- src/bin/pg_dump/t/002_pg_dump.pl| 19 src/bin/pg_upgrade/t/002_pg_upgrade.pl | 26 ++ src/bin/scripts/clusterdb.c | 4 +- src/bin/scripts/reindexdb.c | 4 +- src/bin/scripts/t/011_clusterdb_all.pl | 14 +++ src/bin/scripts/t/050_dropdb.pl | 9 ++ src/bin/scripts/t/091_reindexdb_all.pl | 14 +++ src/bin/scripts/t/101_vacuumdb_all.pl | 14 +++ src/bin/scripts/vacuumdb.c | 2 +- src/include/catalog/pg_database.h | 20 - src/test/recovery/meson.build | 1 + src/test/recovery/t/037_invalid_database.pl | 133 20 files changed, 421 insertions(+), 28 deletions(-)
pgsql: Handle DROP DATABASE getting interrupted
Handle DROP DATABASE getting interrupted Until now, when DROP DATABASE got interrupted in the wrong moment, the removal of the pg_database row would also roll back, even though some irreversible steps have already been taken. E.g. DropDatabaseBuffers() might have thrown out dirty buffers, or files could have been unlinked. But we continued to allow connections to such a corrupted database. To fix this, mark databases invalid with an in-place update, just before starting to perform irreversible steps. As we can't add a new column in the back branches, we use pg_database.datconnlimit = -2 for this purpose. An invalid database cannot be connected to anymore, but can still be dropped. Unfortunately we can't easily add output to psql's \l to indicate that some database is invalid, it doesn't fit in any of the existing columns. Add tests verifying that a interrupted DROP DATABASE is handled correctly in the backend and in various tools. Reported-by: Evgeny Morozov Author: Andres Freund Reviewed-by: Daniel Gustafsson Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/20230509004637.cgvmfwrbht7xm...@awork3.anarazel.de Discussion: https://postgr.es/m/20230314174521.74jl6ffqsee5m...@awork3.anarazel.de Backpatch: 11-, bug present in all supported versions Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/c66a7d75e652801043ece99b6a8f89fd9513eaaa Modified Files -- doc/src/sgml/catalogs.sgml | 3 +- src/backend/commands/dbcommands.c | 110 ++- src/backend/commands/vacuum.c | 14 +++ src/backend/postmaster/autovacuum.c | 12 +++ src/backend/utils/init/postinit.c | 10 +++ src/bin/pg_amcheck/pg_amcheck.c | 2 +- src/bin/pg_amcheck/t/002_nonesuch.pl| 34 +++ src/bin/pg_dump/pg_dumpall.c| 4 +- src/bin/pg_dump/t/002_pg_dump.pl| 19 src/bin/pg_upgrade/t/002_pg_upgrade.pl | 26 ++ src/bin/scripts/clusterdb.c | 4 +- src/bin/scripts/reindexdb.c | 4 +- src/bin/scripts/t/011_clusterdb_all.pl | 14 +++ src/bin/scripts/t/050_dropdb.pl | 9 ++ src/bin/scripts/t/091_reindexdb_all.pl | 14 +++ src/bin/scripts/t/101_vacuumdb_all.pl | 14 +++ src/bin/scripts/vacuumdb.c | 2 +- src/include/catalog/pg_database.h | 20 - src/test/recovery/meson.build | 1 + src/test/recovery/t/037_invalid_database.pl | 133 20 files changed, 421 insertions(+), 28 deletions(-)
pgsql: Handle DROP DATABASE getting interrupted
Handle DROP DATABASE getting interrupted Until now, when DROP DATABASE got interrupted in the wrong moment, the removal of the pg_database row would also roll back, even though some irreversible steps have already been taken. E.g. DropDatabaseBuffers() might have thrown out dirty buffers, or files could have been unlinked. But we continued to allow connections to such a corrupted database. To fix this, mark databases invalid with an in-place update, just before starting to perform irreversible steps. As we can't add a new column in the back branches, we use pg_database.datconnlimit = -2 for this purpose. An invalid database cannot be connected to anymore, but can still be dropped. Unfortunately we can't easily add output to psql's \l to indicate that some database is invalid, it doesn't fit in any of the existing columns. Add tests verifying that a interrupted DROP DATABASE is handled correctly in the backend and in various tools. Reported-by: Evgeny Morozov Author: Andres Freund Reviewed-by: Daniel Gustafsson Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/20230509004637.cgvmfwrbht7xm...@awork3.anarazel.de Discussion: https://postgr.es/m/20230314174521.74jl6ffqsee5m...@awork3.anarazel.de Backpatch: 11-, bug present in all supported versions Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/034a9fcd2bb8e8ac49165e983a44e8805c8a7c63 Modified Files -- doc/src/sgml/catalogs.sgml | 3 +- src/backend/commands/dbcommands.c | 100 +++--- src/backend/commands/vacuum.c | 14 +++ src/backend/postmaster/autovacuum.c | 12 +++ src/backend/utils/init/postinit.c | 10 ++ src/bin/pg_dump/pg_dumpall.c| 4 +- src/bin/pg_dump/t/002_pg_dump.pl| 21 +++- src/bin/scripts/clusterdb.c | 4 +- src/bin/scripts/reindexdb.c | 4 +- src/bin/scripts/t/011_clusterdb_all.pl | 15 ++- src/bin/scripts/t/050_dropdb.pl | 11 +- src/bin/scripts/t/091_reindexdb_all.pl | 15 ++- src/bin/scripts/t/101_vacuumdb_all.pl | 15 ++- src/bin/scripts/vacuumdb.c | 2 +- src/include/catalog/pg_database.h | 20 +++- src/test/recovery/t/037_invalid_database.pl | 157 16 files changed, 380 insertions(+), 27 deletions(-)
pgsql: Release lock after encountering bogs row in vac_truncate_clog()
Release lock after encountering bogs row in vac_truncate_clog() When vac_truncate_clog() encounters bogus datfrozenxid / datminmxid values, it returns early. Unfortunately, until now, it did not release WrapLimitsVacuumLock. If the backend later tries to acquire WrapLimitsVacuumLock, the session / autovacuum worker hangs in an uncancellable way. Similarly, other sessions will hang waiting for the lock. However, if the backend holding the lock exited or errored out for some reason, the lock was released. The bug was introduced as a side effect of 566372b3d643. It is interesting that there are no production reports of this problem. That is likely due to a mix of bugs leading to bogus values having gotten less common, process exit releasing locks and instances of hangs being hard to debug for "normal" users. Discussion: https://postgr.es/m/20230621221208.vhsqgduwfpzwx...@awork3.anarazel.de Branch -- master Details --- https://git.postgresql.org/pg/commitdiff/83ecfa9fad11448af2cbac6c9f2a03507e6317cf Modified Files -- src/backend/commands/vacuum.c | 4 1 file changed, 4 insertions(+)
pgsql: Handle DROP DATABASE getting interrupted
Handle DROP DATABASE getting interrupted Until now, when DROP DATABASE got interrupted in the wrong moment, the removal of the pg_database row would also roll back, even though some irreversible steps have already been taken. E.g. DropDatabaseBuffers() might have thrown out dirty buffers, or files could have been unlinked. But we continued to allow connections to such a corrupted database. To fix this, mark databases invalid with an in-place update, just before starting to perform irreversible steps. As we can't add a new column in the back branches, we use pg_database.datconnlimit = -2 for this purpose. An invalid database cannot be connected to anymore, but can still be dropped. Unfortunately we can't easily add output to psql's \l to indicate that some database is invalid, it doesn't fit in any of the existing columns. Add tests verifying that a interrupted DROP DATABASE is handled correctly in the backend and in various tools. Reported-by: Evgeny Morozov Author: Andres Freund Reviewed-by: Daniel Gustafsson Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/20230509004637.cgvmfwrbht7xm...@awork3.anarazel.de Discussion: https://postgr.es/m/20230314174521.74jl6ffqsee5m...@awork3.anarazel.de Backpatch: 11-, bug present in all supported versions Branch -- REL_13_STABLE Details --- https://git.postgresql.org/pg/commitdiff/81ce67e3c7d5f08adb615806453567ee142f Modified Files -- doc/src/sgml/catalogs.sgml | 3 +- src/backend/commands/dbcommands.c | 100 +++--- src/backend/commands/vacuum.c | 14 +++ src/backend/postmaster/autovacuum.c | 12 +++ src/backend/utils/init/postinit.c | 10 ++ src/bin/pg_dump/pg_dumpall.c| 4 +- src/bin/pg_dump/t/002_pg_dump.pl| 21 +++- src/bin/scripts/clusterdb.c | 4 +- src/bin/scripts/reindexdb.c | 4 +- src/bin/scripts/t/011_clusterdb_all.pl | 15 ++- src/bin/scripts/t/050_dropdb.pl | 11 +- src/bin/scripts/t/091_reindexdb_all.pl | 15 ++- src/bin/scripts/t/101_vacuumdb_all.pl | 15 ++- src/bin/scripts/vacuumdb.c | 2 +- src/include/catalog/pg_database.h | 20 +++- src/test/recovery/t/037_invalid_database.pl | 157 16 files changed, 380 insertions(+), 27 deletions(-)
pgsql: Release lock after encountering bogs row in vac_truncate_clog()
Release lock after encountering bogs row in vac_truncate_clog() When vac_truncate_clog() encounters bogus datfrozenxid / datminmxid values, it returns early. Unfortunately, until now, it did not release WrapLimitsVacuumLock. If the backend later tries to acquire WrapLimitsVacuumLock, the session / autovacuum worker hangs in an uncancellable way. Similarly, other sessions will hang waiting for the lock. However, if the backend holding the lock exited or errored out for some reason, the lock was released. The bug was introduced as a side effect of 566372b3d643. It is interesting that there are no production reports of this problem. That is likely due to a mix of bugs leading to bogus values having gotten less common, process exit releasing locks and instances of hangs being hard to debug for "normal" users. Discussion: https://postgr.es/m/20230621221208.vhsqgduwfpzwx...@awork3.anarazel.de Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/e246fd42363fcfd61cb98fb338c307e52747973f Modified Files -- src/backend/commands/vacuum.c | 4 1 file changed, 4 insertions(+)
pgsql: Handle DROP DATABASE getting interrupted
Handle DROP DATABASE getting interrupted Until now, when DROP DATABASE got interrupted in the wrong moment, the removal of the pg_database row would also roll back, even though some irreversible steps have already been taken. E.g. DropDatabaseBuffers() might have thrown out dirty buffers, or files could have been unlinked. But we continued to allow connections to such a corrupted database. To fix this, mark databases invalid with an in-place update, just before starting to perform irreversible steps. As we can't add a new column in the back branches, we use pg_database.datconnlimit = -2 for this purpose. An invalid database cannot be connected to anymore, but can still be dropped. Unfortunately we can't easily add output to psql's \l to indicate that some database is invalid, it doesn't fit in any of the existing columns. Add tests verifying that a interrupted DROP DATABASE is handled correctly in the backend and in various tools. Reported-by: Evgeny Morozov Author: Andres Freund Reviewed-by: Daniel Gustafsson Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/20230509004637.cgvmfwrbht7xm...@awork3.anarazel.de Discussion: https://postgr.es/m/20230314174521.74jl6ffqsee5m...@awork3.anarazel.de Backpatch: 11-, bug present in all supported versions Branch -- REL_14_STABLE Details --- https://git.postgresql.org/pg/commitdiff/d11efe83038538a7b5169c679ceee457f5753877 Modified Files -- doc/src/sgml/catalogs.sgml | 3 +- src/backend/commands/dbcommands.c | 100 +++--- src/backend/commands/vacuum.c | 14 +++ src/backend/postmaster/autovacuum.c | 12 +++ src/backend/utils/init/postinit.c | 10 ++ src/bin/pg_amcheck/pg_amcheck.c | 2 +- src/bin/pg_amcheck/t/002_nonesuch.pl| 36 ++- src/bin/pg_dump/pg_dumpall.c| 4 +- src/bin/pg_dump/t/002_pg_dump.pl| 21 +++- src/bin/scripts/clusterdb.c | 4 +- src/bin/scripts/reindexdb.c | 4 +- src/bin/scripts/t/011_clusterdb_all.pl | 15 ++- src/bin/scripts/t/050_dropdb.pl | 11 +- src/bin/scripts/t/091_reindexdb_all.pl | 15 ++- src/bin/scripts/t/101_vacuumdb_all.pl | 15 ++- src/bin/scripts/vacuumdb.c | 2 +- src/include/catalog/pg_database.h | 20 +++- src/test/recovery/t/037_invalid_database.pl | 157 18 files changed, 416 insertions(+), 29 deletions(-)
pgsql: Handle DROP DATABASE getting interrupted
Handle DROP DATABASE getting interrupted Until now, when DROP DATABASE got interrupted in the wrong moment, the removal of the pg_database row would also roll back, even though some irreversible steps have already been taken. E.g. DropDatabaseBuffers() might have thrown out dirty buffers, or files could have been unlinked. But we continued to allow connections to such a corrupted database. To fix this, mark databases invalid with an in-place update, just before starting to perform irreversible steps. As we can't add a new column in the back branches, we use pg_database.datconnlimit = -2 for this purpose. An invalid database cannot be connected to anymore, but can still be dropped. Unfortunately we can't easily add output to psql's \l to indicate that some database is invalid, it doesn't fit in any of the existing columns. Add tests verifying that a interrupted DROP DATABASE is handled correctly in the backend and in various tools. Reported-by: Evgeny Morozov Author: Andres Freund Reviewed-by: Daniel Gustafsson Reviewed-by: Thomas Munro Discussion: https://postgr.es/m/20230509004637.cgvmfwrbht7xm...@awork3.anarazel.de Discussion: https://postgr.es/m/20230314174521.74jl6ffqsee5m...@awork3.anarazel.de Backpatch: 11-, bug present in all supported versions Branch -- REL_11_STABLE Details --- https://git.postgresql.org/pg/commitdiff/1c38e7ae17b68a78dcef6a81be9fcf05ba91b374 Modified Files -- doc/src/sgml/catalogs.sgml | 3 +- src/backend/commands/dbcommands.c | 102 +++--- src/backend/commands/vacuum.c | 14 +++ src/backend/postmaster/autovacuum.c | 12 +++ src/backend/utils/init/postinit.c | 10 ++ src/bin/pg_dump/pg_dumpall.c| 4 +- src/bin/pg_dump/t/002_pg_dump.pl| 21 +++- src/bin/scripts/clusterdb.c | 4 +- src/bin/scripts/reindexdb.c | 4 +- src/bin/scripts/t/011_clusterdb_all.pl | 15 ++- src/bin/scripts/t/050_dropdb.pl | 11 +- src/bin/scripts/t/091_reindexdb_all.pl | 15 ++- src/bin/scripts/t/101_vacuumdb_all.pl | 15 ++- src/bin/scripts/vacuumdb.c | 2 +- src/include/catalog/pg_database.h | 21 +++- src/test/recovery/t/037_invalid_database.pl | 157 16 files changed, 383 insertions(+), 27 deletions(-)
pgsql: Release lock after encountering bogs row in vac_truncate_clog()
Release lock after encountering bogs row in vac_truncate_clog() When vac_truncate_clog() encounters bogus datfrozenxid / datminmxid values, it returns early. Unfortunately, until now, it did not release WrapLimitsVacuumLock. If the backend later tries to acquire WrapLimitsVacuumLock, the session / autovacuum worker hangs in an uncancellable way. Similarly, other sessions will hang waiting for the lock. However, if the backend holding the lock exited or errored out for some reason, the lock was released. The bug was introduced as a side effect of 566372b3d643. It is interesting that there are no production reports of this problem. That is likely due to a mix of bugs leading to bogus values having gotten less common, process exit releasing locks and instances of hangs being hard to debug for "normal" users. Discussion: https://postgr.es/m/20230621221208.vhsqgduwfpzwx...@awork3.anarazel.de Branch -- REL_12_STABLE Details --- https://git.postgresql.org/pg/commitdiff/7aec84e4c2652d9928916330f612de21e40621d8 Modified Files -- src/backend/commands/vacuum.c | 4 1 file changed, 4 insertions(+)
