RE: refactoring basebackup.c

2022-02-11 Thread Shinoda, Noriyoshi (PN Japan FSIP)
Hi, Hackers. Thank you for developing a great feature. The current help message shown below does not seem to be able to specify the 'client-' or 'server-' for lz4 compression. --compress = {[{client, server}-]gzip, lz4, none}[:LEVEL] The attached small patch fixes the help message as follows:

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-11 Thread Peter Geoghegan
On Sat, Jan 29, 2022 at 8:42 PM Peter Geoghegan wrote: > Attached is v7, a revision that overhauls the algorithm that decides > what to freeze. I'm now calling it block-driven freezing in the commit > message. Also included is a new patch, that makes VACUUM record zero > free space in the FSM for

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Michael Paquier
On Sat, Feb 12, 2022 at 09:49:47AM +0900, Michael Paquier wrote: > I guess that it would be better to revert the TAP test and rework all > that for the time being, then. And this part is done. -- Michael signature.asc Description: PGP signature

Re: Teach pg_receivewal to use lz4 compression

2022-02-11 Thread Michael Paquier
On Fri, Feb 11, 2022 at 10:07:49AM -0500, Robert Haas wrote: > Over in > http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com > I was noticing that CreateWalTarMethod doesn't support LZ4 > compression. It would be nice if it did. I thought maybe the patch on >

Re: Race condition in TransactionIdIsInProgress

2022-02-11 Thread Andres Freund
Hi, On 2022-02-11 16:41:24 -0800, Andres Freund wrote: > FWIW, I've indeed reproduced this fairly easily with such a setup. A pgbench > r/w workload that's been modified to start 70 savepoints at the start shows > > pgbench: error: client 22 script 0 aborted in command 12 query 0: ERROR: >

Re: Replacing TAP test planning with done_testing()

2022-02-11 Thread Julien Rouhaud
Le sam. 12 févr. 2022 à 04:28, Alvaro Herrera a écrit : > On 2022-Feb-11, Tom Lane wrote: > > > Andres Freund writes: > > > > +1 on backpatching. Backpatching tests now is less likely to cause > conflicts, > > > but more likely to fail during tests. > > > > If you've got the energy to do it, +1

Re: PGroonga index-only scan problem with yesterday’s PostgreSQL updates

2022-02-11 Thread Anders Kaseorg
On 2/11/22 15:57, Tom Lane wrote: Possibly same issue I just fixed? https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=e5691cc9170bcd6c684715c2755d919c5a16fea2 Yeah, that does seem to fix my test cases. Thanks! Any chance this will make it into minor releases sooner than

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Michael Paquier
On Fri, Feb 11, 2022 at 12:34:31PM -0500, Tom Lane wrote: > Justin Pryzby writes: >> Or, is it okay to use ABS_SRCDIR? > > Don't see why not --- other tests certainly do. Another solution would be to rely on TESTDIR, which is always set as of the Makefile invocations and vcregress.pl, but

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Michael Paquier
On Fri, Feb 11, 2022 at 10:48:11AM +0100, Christoph Berg wrote: > It never caused any problem in the 12+ years we have been doing this, > but Debian is patching pg_config not to be relocatable since we are > installing into /usr/lib/postgresql/NN /usr/share/postgresql/NN, so > not a single prefix.

Re: Race condition in TransactionIdIsInProgress

2022-02-11 Thread Andres Freund
Hi, On 2022-02-10 22:11:38 -0800, Andres Freund wrote: > Looks lik syncrep will make this a lot worse, because it can drastically > increase the window between the TransactionIdCommitTree() and > ProcArrayEndTransaction() due to the SyncRepWaitForLSN() inbetween. But at > least it might make it

Re: logical decoding and replication of sequences

2022-02-11 Thread Tomas Vondra
On 2/10/22 19:17, Tomas Vondra wrote: > I've polished & pushed the first part adding sequence decoding > infrastructure etc. Attached are the two remaining parts. > > I plan to wait a day or two and then push the test_decoding part. The > last part (for built-in replication) will need more work

Re: PGroonga index-only scan problem with yesterday’s PostgreSQL updates

2022-02-11 Thread Tom Lane
Anders Kaseorg writes: > With yesterday’s release of PostgreSQL 11.15, 12.10, and 13.6 > (presumably 10.20 and 14.2 as well), Zulip’s test suite started failing > with “variable not found in subplan target list” errors from PostgreSQL > on a table that has a PGroonga index. Possibly same

PGroonga index-only scan problem with yesterday’s PostgreSQL updates

2022-02-11 Thread Anders Kaseorg
With yesterday’s release of PostgreSQL 11.15, 12.10, and 13.6 (presumably 10.20 and 14.2 as well), Zulip’s test suite started failing with “variable not found in subplan target list” errors from PostgreSQL on a table that has a PGroonga index. I found the following reproduction recipe from a

Re: support for MERGE

2022-02-11 Thread Justin Pryzby
On Fri, Feb 11, 2022 at 05:25:49PM -0300, Alvaro Herrera wrote: > > I'm not sure git diff --cherry-pick is widely known/used, but I think > > using that relative to master may be good enough. > > I had never heard of git diff --cherry-pick, and the manpages I found > don't document it, so

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-02-11 Thread Andrew Dunstan
On 2/6/22 10:45, Robert Haas wrote: > For what it's worth, I am generally in favor of having something like > this in PostgreSQL. I think it's wrong of us to continue assuming that > everyone has command-line access. Even when that's true, it's not > necessarily convenient. If you choose to use

Re: Fix overflow in DecodeInterval

2022-02-11 Thread Joseph Koshakow
Tom Lane writes: > Joseph Koshakow writes: > > The attached patch fixes an overflow bug in DecodeInterval when applying > > the units week, decade, century, and millennium. The overflow check logic > > was modelled after the overflow check at the beginning of `int > > tm2interval(struct pg_tm

Re: Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2022-02-11 Thread Thomas Munro
On Thu, Jan 6, 2022 at 10:23 AM Robert Haas wrote: > That's fixed now. So what should we do about this patch? This is a > bug, so it would be nice to do *something*. I don't really like the > fact that this makes the behavior contingent on USE_ASSERT_CHECKING, > and I suggest that you make a new

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 4:08 PM Alvaro Herrera wrote: > It seems you're thinking deciding what to do based on an option that > gets a boolean argument. But what about making the argument be an enum? > For example > > CREATE DATABASE ... WITH (STRATEGY = LOG); -- default if option is >

Re: Merging statistics from children instead of re-sampling everything

2022-02-11 Thread Robert Haas
On Wed, Jun 30, 2021 at 11:15 AM Tomas Vondra wrote: > You're right maintaining a per-partition samples and merging those might > solve (or at least reduce) some of the problems, e.g. eliminating most > of the I/O that'd be needed for sampling. And yeah, it's not entirely > clear how to merge

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Andrew Dunstan
On 2/11/22 15:47, Robert Haas wrote: > On Fri, Feb 11, 2022 at 3:40 PM Andrew Dunstan wrote: >> I'm not really sure any single parameter name is going to capture the >> subtlety involved here. > I mean to some extent that's inevitable, but it's not a reason not to > do the best we can. True.

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Alvaro Herrera
On 2022-Feb-11, Robert Haas wrote: > What I find difficult about doing that is that this is all a bunch of > technical details that users may have difficulty understanding. If we > say WAL_LOG or WAL_LOG_DATA, a reasonably but not incredibly > well-informed user will assume that skipping WAL is

Re: Per-table storage parameters for TableAM/IndexAM extensions

2022-02-11 Thread Robert Haas
On Wed, Dec 29, 2021 at 12:08 PM Sadhuprasad Patro wrote: > Open Question: When a user changes AM, then what should be the > behavior for not supported storage options? Should we drop the options > and go with only system storage options? > Or throw an error, in which case the user has to clean

Re: Fix overflow in DecodeInterval

2022-02-11 Thread Tom Lane
Joseph Koshakow writes: > The attached patch fixes an overflow bug in DecodeInterval when applying > the units week, decade, century, and millennium. The overflow check logic > was modelled after the overflow check at the beginning of `int > tm2interval(struct pg_tm *tm, fsec_t fsec, Interval

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 1:32 PM Bruce Momjian wrote: > > Yeah, maybe. But it's not clear to me with that kind of naming whether > > TRUE or FALSE would be the existing behavior? One version logs a > > single record for the whole database, and the other logs a record per > > database block.

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 3:40 PM Andrew Dunstan wrote: > I'm not really sure any single parameter name is going to capture the > subtlety involved here. I mean to some extent that's inevitable, but it's not a reason not to do the best we can. -- Robert Haas EDB: http://www.enterprisedb.com

Re: support for CREATE MODULE

2022-02-11 Thread Robert Haas
On Thu, Feb 10, 2022 at 4:17 PM Alvaro Herrera wrote: > On 2022-Feb-04, Tom Lane wrote: > > If we invent modules I think they need to work more like extensions > > naming-wise, ie they group objects but have no effect for naming. > > I think modules are an orthogonal concept to extensions, and

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Andrew Dunstan
On 2/11/22 13:32, Bruce Momjian wrote: > On Fri, Feb 11, 2022 at 01:18:58PM -0500, Robert Haas wrote: >> On Fri, Feb 11, 2022 at 12:50 PM Bruce Momjian wrote: >>> On Fri, Feb 11, 2022 at 12:35:50PM -0500, Robert Haas wrote: How about something like LOG_AS_CLONE? That makes it clear, I

Re: Replacing TAP test planning with done_testing()

2022-02-11 Thread Alvaro Herrera
On 2022-Feb-11, Tom Lane wrote: > Andres Freund writes: > > +1 on backpatching. Backpatching tests now is less likely to cause > > conflicts, > > but more likely to fail during tests. > > If you've got the energy to do it, +1 for backpatching. I agree > with Michael's opinion that doing so

Re: support for MERGE

2022-02-11 Thread Alvaro Herrera
On 2022-Feb-11, Justin Pryzby wrote: > Because I put your patch on top of some other branch with the CI coverage (and > other stuff). Ah, that makes sense. > But it has to figure out where the branch "starts". Which I did by looking at > "git diff --cherry-pick origin..." > > I'm not sure git

Re: Replacing TAP test planning with done_testing()

2022-02-11 Thread Tom Lane
Andres Freund writes: > On 2022-02-11 21:04:53 +0100, Daniel Gustafsson wrote: >> I opted out of backpatching for now, to solicit more comments on that. It's >> not a bugfix, but it's also not affecting the compiled bits that we ship, so >> I >> think there's a case to be made both for and

Re: Replacing TAP test planning with done_testing()

2022-02-11 Thread Andres Freund
On 2022-02-11 21:04:53 +0100, Daniel Gustafsson wrote: > I opted out of backpatching for now, to solicit more comments on that. It's > not a bugfix, but it's also not affecting the compiled bits that we ship, so I > think there's a case to be made both for and against a backpatch. Looking at >

Re: Replacing TAP test planning with done_testing()

2022-02-11 Thread Daniel Gustafsson
> On 10 Feb 2022, at 01:58, Michael Paquier wrote: >>> Daniel Gustafsson writes: The attached patch removes all Test::More planning and instead ensures that all tests conclude with a done_testing() call. Pushed to master now with a few more additional hunks fixing test changes

Re: postgres_fdw: using TABLESAMPLE to collect remote sample

2022-02-11 Thread Tomas Vondra
On 2/11/22 19:13, Robert Haas wrote: > On Fri, Feb 11, 2022 at 1:06 PM Tom Lane wrote: >> While I'm not opposed to moving those goalposts at some point, >> I think pushing them all the way up to 9.5 for this one easily-fixed >> problem is not very reasonable. >> >> Given other recent discussion,

Fix overflow in DecodeInterval

2022-02-11 Thread Joseph Koshakow
The attached patch fixes an overflow bug in DecodeInterval when applying the units week, decade, century, and millennium. The overflow check logic was modelled after the overflow check at the beginning of `int tm2interval(struct pg_tm *tm, fsec_t fsec, Interval *span);` in timestamp.c. This is my

Re: Race condition in TransactionIdIsInProgress

2022-02-11 Thread Andres Freund
Hi, On 2022-02-11 13:48:59 +, Simon Riggs wrote: > On Fri, 11 Feb 2022 at 08:48, Simon Riggs > wrote: > > > > On Fri, 11 Feb 2022 at 06:11, Andres Freund wrote: > > > > > Looks lik syncrep will make this a lot worse, because it can drastically > > > increase the window between the

Re: support for MERGE

2022-02-11 Thread Justin Pryzby
On Fri, Feb 11, 2022 at 03:21:43PM -0300, Alvaro Herrera wrote: > On 2022-Jan-28, Justin Pryzby wrote: > > Have you looked at code coverage ? I have an experimental patch to add > > that to > > cirrus, and ran it with this patch; visible here: > > https://cirrus-ci.com/task/6362512059793408 > >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Bruce Momjian
On Fri, Feb 11, 2022 at 01:18:58PM -0500, Robert Haas wrote: > On Fri, Feb 11, 2022 at 12:50 PM Bruce Momjian wrote: > > On Fri, Feb 11, 2022 at 12:35:50PM -0500, Robert Haas wrote: > > > How about something like LOG_AS_CLONE? That makes it clear, I hope, > > > that we're logging it a different

Re: support for MERGE

2022-02-11 Thread Alvaro Herrera
On 2022-Jan-28, Justin Pryzby wrote: > Have you looked at code coverage ? I have an experimental patch to add that > to > cirrus, and ran it with this patch; visible here: > https://cirrus-ci.com/task/6362512059793408 Ah, thanks, this is useful. I think it is showing that the new code is

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 12:50 PM Bruce Momjian wrote: > On Fri, Feb 11, 2022 at 12:35:50PM -0500, Robert Haas wrote: > > How about something like LOG_AS_CLONE? That makes it clear, I hope, > > that we're logging it a different way, but that method of logging it > > is different in each case.

Re: postgres_fdw: using TABLESAMPLE to collect remote sample

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 1:06 PM Tom Lane wrote: > While I'm not opposed to moving those goalposts at some point, > I think pushing them all the way up to 9.5 for this one easily-fixed > problem is not very reasonable. > > Given other recent discussion, an argument could be made for moving > the

Re: postgres_fdw: using TABLESAMPLE to collect remote sample

2022-02-11 Thread Tom Lane
Robert Haas writes: > I think it's going to be necessary to compromise on that at some > point. Sure. The existing postgres_fdw documentation [1] already addresses this issue: postgres_fdw can be used with remote servers dating back to PostgreSQL 8.3. Read-only capability is available

Re: O(n) tasks cause lengthy startups and checkpoints

2022-02-11 Thread Nathan Bossart
Here is a rebased patch set. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com >From 506aa95dd77f16dc64d7fe9c52ca67dd3c10212e Mon Sep 17 00:00:00 2001 From: Nathan Bossart Date: Wed, 5 Jan 2022 19:24:22 + Subject: [PATCH v4 1/8] Introduce custodian. The custodian process is a

[PATCH] Support % wildcard in extension upgrade filenames

2022-02-11 Thread Sandro Santilli
At PostGIS we've been thinking of ways to have better support, from PostgreSQL proper, to our upgrade strategy, which has always consisted in a SINGLE upgrade file good for upgrading from any older version. The current lack of such support for EXTENSIONs forced us to install a lot of files on

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Bruce Momjian
On Fri, Feb 11, 2022 at 12:35:50PM -0500, Robert Haas wrote: > How about something like LOG_AS_CLONE? That makes it clear, I hope, > that we're logging it a different way, but that method of logging it > is different in each case. You'd still have to read the documentation > to find out what it

Re: postgres_fdw: using TABLESAMPLE to collect remote sample

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 12:39 PM Tom Lane wrote: > Tomas Vondra writes: > > So here we go. The patch does a very simple thing - it uses TABLESAMPLE > > to collect/transfer just a small sample from the remote node, saving > > both CPU and network. > > This is great if the remote end has

Re: postgres_fdw: using TABLESAMPLE to collect remote sample

2022-02-11 Thread Tom Lane
Tomas Vondra writes: > So here we go. The patch does a very simple thing - it uses TABLESAMPLE > to collect/transfer just a small sample from the remote node, saving > both CPU and network. This is great if the remote end has TABLESAMPLE, but pre-9.5 servers don't, and postgres_fdw is supposed

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 12:11 PM Andrew Dunstan wrote: > The last one at least has the advantage that it doesn't invent yet > another keyword. We don't need a new keyword for this as long as it lexes as one token, because createdb_opt_name accepts IDENT. So I think we should focus on trying to

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Tom Lane
Justin Pryzby writes: > Or, is it okay to use ABS_SRCDIR? Don't see why not --- other tests certainly do. regards, tom lane

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Justin Pryzby
On Fri, Feb 11, 2022 at 10:41:27AM -0500, Tom Lane wrote: > Justin Pryzby writes: > > On Fri, Feb 11, 2022 at 09:59:55AM -0500, Tom Lane wrote: > >> This seems like a pretty bad idea even if it weren't failing outright. > >> We should be examining the version of the file that's in the source > >>

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-02-11 Thread Andrew Dunstan
On 2/10/22 07:32, Dilip Kumar wrote: > On Wed, Feb 9, 2022 at 9:31 PM Robert Haas wrote: >> On Wed, Feb 9, 2022 at 10:59 AM Dilip Kumar wrote: >>> On Wed, Feb 9, 2022 at 9:25 PM Andrew Dunstan wrote: On 6/16/21 03:52, Dilip Kumar wrote: > On Tue, Jun 15, 2021 at 7:01 PM Andrew

Re: the build farm is ok, but not the hippopotamus (or the jay)

2022-02-11 Thread Tomas Vondra
On 2/11/22 16:16, Robert Haas wrote: Hi, I was looking over the buildfarm results yesterday and this morning, and things seem to be mostly green, but not entirely. 'hippopotamus' was failing yesterday with strange LDAP-related errors, and is now complaining about pgsql-Git-Dirty, which I assume

postgres_fdw: using TABLESAMPLE to collect remote sample

2022-02-11 Thread Tomas Vondra
Hi, here's a small patch modifying postgres_fdw to use TABLESAMPLE to collect sample when running ANALYZE on a foreign table. Currently the sampling happens locally, i.e. we fetch all data from the remote server and then pick rows. But that's hugely expensive for large relations and/or with

Re: Assertion failure in WaitForWALToBecomeAvailable state machine

2022-02-11 Thread Bharath Rupireddy
On Fri, Feb 11, 2022 at 6:31 PM Dilip Kumar wrote: > > On Fri, Feb 11, 2022 at 6:22 PM Bharath Rupireddy > wrote: > > > > On Fri, Feb 11, 2022 at 3:33 PM Dilip Kumar wrote: > > > > > > IIUC, the issue can happen while the walreceiver failed to get WAL > > from primary for whatever reasons and

Re: pg_receivewal.exe unhandled exception in zlib1.dll

2022-02-11 Thread Andres Freund
Hi, On 2022-02-11 16:33:11 +0100, Juan José Santamaría Flecha wrote: > If you execute pg_receivewal.exe with the option "--compression-method > gzip" it will fail with no error. You can see an error in the db log: > > 2022-02-10 11:46:32.725 CET [11664][walsender] [pg_receivewal][3/0:0] LOG: >

Re: refactoring basebackup.c

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 10:29 AM Justin Pryzby wrote: > FYI: there's a couple typos in the last 2 patches. Hmm. OK. But I don't consider "can be optionally specified" incorrect or worse than "can optionally be specified". I do agree that spelling words correctly is a good idea. -- Robert Haas

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Tom Lane
Justin Pryzby writes: > On Fri, Feb 11, 2022 at 09:59:55AM -0500, Tom Lane wrote: >> This seems like a pretty bad idea even if it weren't failing outright. >> We should be examining the version of the file that's in the source >> tree; the one in the installation tree might have version-skew >>

Re: OpenSSL conflicts with wincrypt.h

2022-02-11 Thread Tom Lane
=?UTF-8?Q?Juan_Jos=C3=A9_Santamar=C3=ADa_Flecha?= writes: > There is a comment in 'src/backend/libpq/be-secure-openssl.c' addressing > this issue, but I have to explicitly undefine X509_NAME. Please find > attached a patch for so. Um ... why? Shouldn't the #undef in the OpenSSL headers take

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Justin Pryzby
On Fri, Feb 11, 2022 at 09:59:55AM -0500, Tom Lane wrote: > Christoph Berg writes: > > this test is failing at Debian package compile time: > > Could not open /usr/share/postgresql/15/postgresql.conf.sample: No such > > file or directory at t/003_check_guc.pl line 47. > > > So it's trying to

pg_receivewal.exe unhandled exception in zlib1.dll

2022-02-11 Thread Juan José Santamaría Flecha
Hello, If you execute pg_receivewal.exe with the option "--compression-method gzip" it will fail with no error. You can see an error in the db log: 2022-02-10 11:46:32.725 CET [11664][walsender] [pg_receivewal][3/0:0] LOG: could not receive data from client: An existing connection was forcibly

OpenSSL conflicts with wincrypt.h

2022-02-11 Thread Juan José Santamaría Flecha
Hello, When building Postgres using MSVC with Kerberos Version 4.1 and OpenSSL 1.1.1l (both of them, using only one will raise no errors), I see errors like: "C:\postgres\pgsql.sln" (default target) (1) -> "C:\postgres\postgres.vcxproj" (default target) (2) -> (ClCompile target) ->

Re: refactoring basebackup.c

2022-02-11 Thread Justin Pryzby
On Fri, Feb 11, 2022 at 08:35:25PM +0530, Jeevan Ladhe wrote: > Thanks Robert for the bravity :-) FYI: there's a couple typos in the last 2 patches. I added them to my typos branch; feel free to wait until April if you'd prefer to see them fixed in bulk. diff --git

the build farm is ok, but not the hippopotamus (or the jay)

2022-02-11 Thread Robert Haas
Hi, I was looking over the buildfarm results yesterday and this morning, and things seem to be mostly green, but not entirely. 'hippopotamus' was failing yesterday with strange LDAP-related errors, and is now complaining about pgsql-Git-Dirty, which I assume means that someone is doing something

Re: Merging statistics from children instead of re-sampling everything

2022-02-11 Thread Tomas Vondra
On 2/11/22 05:29, Andrey V. Lepikhov wrote: On 2/11/22 03:37, Tomas Vondra wrote: That being said, this thread was not really about foreign partitions, but about re-analyzing inheritance trees in general. And sampling foreign partitions doesn't really solve that - we'll still do the sampling

Re: Teach pg_receivewal to use lz4 compression

2022-02-11 Thread Robert Haas
On Thu, Nov 4, 2021 at 10:47 PM Michael Paquier wrote: > Indeed. My rebase was a bit sloppy here. Hi! Over in http://postgr.es/m/CA+TgmoYUDEJga2qV_XbAZ=pgebaosgfmzz6ac4_srwom_+u...@mail.gmail.com I was noticing that CreateWalTarMethod doesn't support LZ4 compression. It would be nice if it

Re: refactoring basebackup.c

2022-02-11 Thread Jeevan Ladhe
Thanks Robert for the bravity :-) Regards, Jeevan Ladhe On Fri, 11 Feb 2022, 20:31 Robert Haas, wrote: > On Fri, Feb 11, 2022 at 7:20 AM Dipesh Pandit > wrote: > > > Sure, please find the rebased patch attached. > > > > Thanks, I have validated v2 patch on top of rebased patch. > > I'm still

Re: refactoring basebackup.c

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 7:20 AM Dipesh Pandit wrote: > > Sure, please find the rebased patch attached. > > Thanks, I have validated v2 patch on top of rebased patch. I'm still feeling brave, so I committed this too after fixing a few things. In the process I noticed that we don't have support

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Tom Lane
Christoph Berg writes: > this test is failing at Debian package compile time: > Could not open /usr/share/postgresql/15/postgresql.conf.sample: No such file > or directory at t/003_check_guc.pl line 47. > So it's trying to read from /usr/share/postgresql which doesn't exist > yet at build time.

Re: Synchronizing slots from primary to standby

2022-02-11 Thread Peter Eisentraut
On 05.02.22 20:59, Andres Freund wrote: On 2022-01-03 14:46:52 +0100, Peter Eisentraut wrote: From ec00dc6ab8bafefc00e9b1c78ac9348b643b8a87 Mon Sep 17 00:00:00 2001 From: Peter Eisentraut Date: Mon, 3 Jan 2022 14:43:36 +0100 Subject: [PATCH v3] Synchronize logical replication slots from

Re: Accommodate startup process in a separate ProcState array slot instead of in MaxBackends slots.

2022-02-11 Thread Yura Sokolov
В Сб, 16/10/2021 в 16:37 +0530, Bharath Rupireddy пишет: > On Thu, Oct 14, 2021 at 10:56 AM Fujii Masao > wrote: > > On 2021/10/12 15:46, Bharath Rupireddy wrote: > > > On Tue, Oct 12, 2021 at 5:37 AM Fujii Masao > > > wrote: > > > > On 2021/10/12 4:07, Bharath Rupireddy wrote: > > > > > Hi, >

Re: Synchronizing slots from primary to standby

2022-02-11 Thread Peter Eisentraut
On 10.02.22 22:47, Bruce Momjian wrote: On Tue, Feb 8, 2022 at 08:27:32PM +0530, Ashutosh Sharma wrote: Which means that if e.g. the standby_slot_names GUC differs from synchronize_slot_names on the physical replica, the slots synchronized on the physical replica are not going to be valid. Or

Re: refactoring basebackup.c

2022-02-11 Thread Robert Haas
On Fri, Feb 11, 2022 at 5:58 AM Jeevan Ladhe wrote: > >Jeevan, Your v12 patch does not apply on HEAD, it requires a > rebase. > > Sure, please find the rebased patch attached. It's Friday today, but I'm feeling brave, and it's still morning here, so ... committed. -- Robert Haas EDB:

Re: Race condition in TransactionIdIsInProgress

2022-02-11 Thread Simon Riggs
On Fri, 11 Feb 2022 at 08:48, Simon Riggs wrote: > > On Fri, 11 Feb 2022 at 06:11, Andres Freund wrote: > > > Looks lik syncrep will make this a lot worse, because it can drastically > > increase the window between the TransactionIdCommitTree() and > > ProcArrayEndTransaction() due to the

Re: Identify missing publications from publisher while create/alter subscription.

2022-02-11 Thread Ashutosh Sharma
I have spent little time looking at the latest patch. The patch looks to be in good shape as it has already been reviewed by many people here, although I did get some comments. Please take a look and let me know your thoughts. + /* Try to connect to the publisher. */ + wrconn =

Re: Assertion failure in WaitForWALToBecomeAvailable state machine

2022-02-11 Thread Dilip Kumar
On Fri, Feb 11, 2022 at 6:22 PM Bharath Rupireddy wrote: > > On Fri, Feb 11, 2022 at 3:33 PM Dilip Kumar wrote: > > > IIUC, the issue can happen while the walreceiver failed to get WAL > from primary for whatever reasons and its status is not > WALRCV_STOPPING or WALRCV_STOPPED, and the startup

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-11 Thread Etsuro Fujita
On Tue, Feb 8, 2022 at 3:49 AM Fujii Masao wrote: > Here are the review comments for 0001 patch. > > I got the following compiler warning. > > [16:58:07.120] connection.c: In function ‘pgfdw_finish_pre_commit_cleanup’: > [16:58:07.120] connection.c:1726:4: error: ISO C90 forbids mixed

Re: Assertion failure in WaitForWALToBecomeAvailable state machine

2022-02-11 Thread Bharath Rupireddy
On Fri, Feb 11, 2022 at 3:33 PM Dilip Kumar wrote: > > Hi, > > The problem is that whenever we are going for streaming we always set > XLogCtl->InstallXLogFileSegmentActive to true, but while switching > from streaming to archive we do not always reset it so it hits > assertion in some cases.

Re: Database-level collation version tracking

2022-02-11 Thread Julien Rouhaud
On Fri, Feb 11, 2022 at 12:07:02PM +0100, Peter Eisentraut wrote: > On 10.02.22 12:08, Julien Rouhaud wrote: > > > + errhint("Rebuild all objects affected > > > by collation in the template database and run " > > > +

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-02-11 Thread Matthias van de Meent
Hi, Today I noticed the inefficiencies of our dead tuple storage once again, and started theorizing about a better storage method; which is when I remembered that this thread exists, and that this thread already has amazing results. Are there any plans to get the results of this thread from PoC

Re: refactoring basebackup.c

2022-02-11 Thread Dipesh Pandit
> Sure, please find the rebased patch attached. Thanks, I have validated v2 patch on top of rebased patch. Thanks, Dipesh

Re: Database-level collation version tracking

2022-02-11 Thread Peter Eisentraut
On 10.02.22 12:08, Julien Rouhaud wrote: +errhint("Rebuild all objects affected by collation in the template database and run " +"ALTER DATABASE %s REFRESH COLLATION VERSION, " +

Re: refactoring basebackup.c

2022-02-11 Thread Jeevan Ladhe
>Jeevan, Your v12 patch does not apply on HEAD, it requires a rebase. Sure, please find the rebased patch attached. Regards, Jeevan On Fri, 11 Feb 2022 at 14:13, Dipesh Pandit wrote: > Hi, > > Thanks for the feedback, I have incorporated the suggestions > and updated a new patch. PFA v2

Re: [BUG]Update Toast data failure in logical replication

2022-02-11 Thread Amit Kapila
On Fri, Feb 11, 2022 at 12:00 PM tanghy.f...@fujitsu.com wrote: > > Attached the patches which fixed the above two comments and the first comment > in > my previous mail [1], the rest is the same as before. > I ran the tests on all branches, they all passed as expected. > Thanks, these look

Re: Condition pushdown: why (=) is pushed down into join, but BETWEEN or >= is not?

2022-02-11 Thread Andy Fan
> >> > So I think knowing what bad it is to have this feature is the key point to >> discussion now. >> >> I re-read the discussion at 2015 [1] and the below topic is added for the above question. Here is the summary for easy discussion. >From planner aspect: > While I've only read your

Assertion failure in WaitForWALToBecomeAvailable state machine

2022-02-11 Thread Dilip Kumar
Hi, The problem is that whenever we are going for streaming we always set XLogCtl->InstallXLogFileSegmentActive to true, but while switching from streaming to archive we do not always reset it so it hits assertion in some cases. Basically we reset it inside XLogShutdownWalRcv() but while

[PATCH] minor reloption regression tests improvement

2022-02-11 Thread Nikolay Shaplov
I'd like to suggest a patch for reloption regression tests. This patch tests case, that can be rarely met in actual life: when reloptions have some illegal option set (as a result of malfunction or extension downgrade or something), and user tries to remove this option by using RESET. Current

Re: pgsql: Add TAP test to automate the equivalent of check_guc

2022-02-11 Thread Christoph Berg
Re: Michael Paquier > Add TAP test to automate the equivalent of check_guc > > src/backend/utils/misc/check_guc is a script that cross-checks the > consistency of the GUCs with postgresql.conf.sample, making sure that Hi, this test is failing at Debian package compile time: 00:55:07 ok 18 -

Re: unlogged sequences

2022-02-11 Thread Peter Eisentraut
On 25.06.19 20:37, Andres Freund wrote: I.e. I think it'd be better if we just added a fork argument to fill_seq_with_data(), and then do something like smgrcreate(srel, INIT_FORKNUM, false); log_smgrcreate(>rd_node, INIT_FORKNUM); fill_seq_with_data(rel, tuple, INIT_FORKNUM); and add a

Re: Race condition in TransactionIdIsInProgress

2022-02-11 Thread Simon Riggs
On Fri, 11 Feb 2022 at 06:11, Andres Freund wrote: > Looks lik syncrep will make this a lot worse, because it can drastically > increase the window between the TransactionIdCommitTree() and > ProcArrayEndTransaction() due to the SyncRepWaitForLSN() inbetween. But at > least it might make it

Re: refactoring basebackup.c

2022-02-11 Thread Dipesh Pandit
Hi, Thanks for the feedback, I have incorporated the suggestions and updated a new patch. PFA v2 patch. > I think similar to bbstreamer_lz4_compressor_content() in > bbstreamer_lz4_decompressor_content() we can change len to avail_in. In bbstreamer_lz4_decompressor_content(), we are modifying