Re: Race conditions in 019_replslot_limit.pl

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 12:15:58 -0800, Andres Freund wrote: > On 2022-02-25 15:07:01 -0500, Tom Lane wrote: > > Andres Freund writes: > > Hmm. Maybe put a couple more debug messages into ReplicationSlotCleanup > > and/or ReplicationSlotDropPtr? It doesn't seem very clear where in that > > sequence

Re: Allow file inclusion in pg_hba and pg_ident files

2022-02-25 Thread Julien Rouhaud
On Sat, Feb 26, 2022 at 03:36:19PM +0900, Michael Paquier wrote: > On Sat, Feb 26, 2022 at 02:27:15PM +0800, Julien Rouhaud wrote: > > > Note that in order to do so we would need to expose quite a lot more about > > hba > > internals, like tokenize_file() and parse_hba_line(), along with structs

Re: Allow file inclusion in pg_hba and pg_ident files

2022-02-25 Thread Michael Paquier
On Sat, Feb 26, 2022 at 02:27:15PM +0800, Julien Rouhaud wrote: > I'm fine with it. Assuming that you meant to move also the underlying > functions that goes with it (fill_hba_line and such), that would end up > removing about 15% of hba.c (after applying 0001, 0002 and 0003). Cool. Thanks for

Re: Commitfest manager for 2022-03

2022-02-25 Thread Julien Rouhaud
On Sat, Feb 26, 2022 at 02:42:33PM +0900, Michael Paquier wrote: > On Fri, Feb 25, 2022 at 01:58:55PM -0600, David Steele wrote: > > On 2/25/22 12:39, Greg Stark wrote: > >> I would like to volunteer. > > > I've been hoping somebody would volunteer, so I'm all in favor of you being > > CF. > >

Re: Allow file inclusion in pg_hba and pg_ident files

2022-02-25 Thread Julien Rouhaud
Hi, On Sat, Feb 26, 2022 at 03:04:43PM +0900, Michael Paquier wrote: > On Wed, Feb 23, 2022 at 09:44:58AM -0800, Nathan Bossart wrote: > > On Wed, Feb 23, 2022 at 12:59:59PM +0800, Julien Rouhaud wrote: > >> 0001 adds a new pg_ident_file_mappings view, which is basically the same as > >>

Re: Allow file inclusion in pg_hba and pg_ident files

2022-02-25 Thread Michael Paquier
On Wed, Feb 23, 2022 at 09:44:58AM -0800, Nathan Bossart wrote: > On Wed, Feb 23, 2022 at 12:59:59PM +0800, Julien Rouhaud wrote: >> 0001 adds a new pg_ident_file_mappings view, which is basically the same as >> pg_hba_file_rules view but for mappings. It's probably already useful, for >>

Re: Frontend error logging style

2022-02-25 Thread Tom Lane
Michael Paquier writes: > Hm. I am not sure to like much this abstraction by having so many > macros that understand the log level through their name. Wouldn't it > be cleaner to have only one pg_log_hint() and one pg_log_detail() with > the log level passed as argument of the macro? Mmm ...

Re: PATCH: add "--config-file=" option to pg_rewind

2022-02-25 Thread Michael Paquier
On Fri, Feb 25, 2022 at 10:35:49AM +0100, Gunnar "Nick" Bluth wrote: > Am 24.02.22 um 14:46 schrieb Daniel Gustafsson: >> Actually, I think this looks like a saner approach. Putting a config setting >> in two place (postgresql.conf and on the commandline for pg_rewind) is a >> recipe >> for them

Re: Commitfest manager for 2022-03

2022-02-25 Thread Michael Paquier
On Fri, Feb 25, 2022 at 01:58:55PM -0600, David Steele wrote: > On 2/25/22 12:39, Greg Stark wrote: >> I would like to volunteer. > > Since I've done the March CF for the last seven years, I thought it would be > good if I chimed in. I've been agonizing a bit because I have travel planned >

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-25 Thread Michael Paquier
On Fri, Feb 25, 2022 at 01:23:49PM -0800, Andres Freund wrote: > Looks to me like authn_id isn't synchronized to parallel workers right now. So > the function will return the wrong thing when executed as part of a parallel > query. FWIW, I am not completely sure what's the use case for being able

Re: Frontend error logging style

2022-02-25 Thread Michael Paquier
On Fri, Feb 25, 2022 at 12:15:25PM -0500, Tom Lane wrote: > I feel that the reasonable alternatives are either to drop the FATAL > log level, or try to make it actually mean something by consistently > using it for errors that are indeed fatal. I had a go at doing the > latter, but eventually

Re: should vacuum's first heap pass be read-only?

2022-02-25 Thread Dilip Kumar
On Fri, Feb 25, 2022 at 10:45 PM Peter Geoghegan wrote: > > On Fri, Feb 25, 2022 at 5:06 AM Dilip Kumar wrote: > > Based on this discussion, IIUC, we are saying that now we will do the > > lazy_scan_heap every time like we are doing now. And we will > > conditionally skip the index vacuum for

Re: Fix typo in logicalfuncs.c - :%s/private date/Private data

2022-02-25 Thread Amit Kapila
On Fri, Feb 25, 2022 at 8:39 PM Bharath Rupireddy wrote: > > Hi, > > Here's a tiny patch to do $subject. > LGTM. I'll push this in some time. -- With Regards, Amit Kapila.

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-25 Thread Amit Kapila
On Sat, Feb 26, 2022 at 3:01 AM Melanie Plageman wrote: > > Since _hash_alloc_buckets() WAL-logs the last page of the > splitpoint, is it safe to skip the smgrimmedsync()? What if the last > page of the splitpoint doesn't end up having any tuples added to it > during the index build and the redo

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2022-02-25 Thread Michael Paquier
On Fri, Feb 25, 2022 at 01:09:53PM -0800, Nathan Bossart wrote: > This one has been quiet for a while. Should we mark it as > returned-with-feedback? Yes, that's my feeling and I got cold feet about this change. This patch would bring some extra visibility for something that's not incorrect

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Amit Kapila
On Fri, Feb 25, 2022 at 7:26 AM Peter Smith wrote: > > Below are my review comments for the v3 patch. > ... > 2. doc/src/sgml/monitoring.sgml > > + > pg_stat_subscription_activitypg_stat_subscription_activity > + One row per subscription, showing statistics about subscription > +

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Amit Kapila
On Thu, Feb 24, 2022 at 9:20 PM Masahiko Sawada wrote: > I have reviewed the latest version and made a few changes along with fixing some of the pending comments by Peter Smith. The changes are as follows: (a) Removed m_databaseid in PgStat_MsgSubscriptionError as that is not required now; (b)

Re: Adding CI to our tree

2022-02-25 Thread Justin Pryzby
This is the other half of my CI patches, which are unrelated to the TAP ones on the other thread. >From 88c01c09ee26db2817629265fc12b2dbcd8c9a91 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 17 Jan 2022 00:53:04 -0600 Subject: [PATCH 1/7] cirrus: include hints how to install OS

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Peter Geoghegan
On Fri, Feb 25, 2022 at 3:26 PM Andres Freund wrote: > freeze_required_limit, freeze_desired_limit? Or s/limit/cutoff/? Or > s/limit/below/? I kind of like below because that answers < vs <= which I find > hard to remember around freezing. I like freeze_required_limit the most. > That may be

Re: convert libpq uri-regress tests to tap test

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 09:56:47 -0800, Andres Freund wrote: > On 2022-02-24 08:46:23 -0800, Andres Freund wrote: > > I'm mildly inclined to only do 0001 and 0002 for now. We'd not loose msvc > > coverage, because it already doesn't build the test. Once we've ironed that > > stuff out, we could do

Re: set TESTDIR from perl rather than Makefile

2022-02-25 Thread Justin Pryzby
On Sun, Feb 20, 2022 at 04:39:08PM -0800, Andres Freund wrote: > On 2022-02-19 17:53:09 -0600, Justin Pryzby wrote: > > I also meant to also attach it. > > Is the patch actually independent of the other patches in your stack? Yes - I rearranged it that way for this thread. However, it's best

Re: Mingw task for Cirrus CI

2022-02-25 Thread Andres Freund
Hi, Andrew, CCIng you both because you might be interested in the CI bit, and because you might know the answer. On 2022-02-25 19:44:27 +0300, Melih Mutlu wrote: > I've been working on adding Windows+MinGW environment into cirrus-ci tasks > (discussion about ci is here [1]). > It uses MSYS2 to

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Peter Geoghegan
On Fri, Feb 25, 2022 at 3:48 PM Andres Freund wrote: > I don't see why it matters that OldestXmin and OldestMxact are computed at the > same time? It's a question of the workload, not vacuum algorithm. I think it's both. > OldestMxact inherently lags OldestXmin. OldestMxact can only advance

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 15:28:17 -0800, Peter Geoghegan wrote: > But why should we ever get to the FreezeMultiXactId() loop with the > stuff from 0002 in place? The whole purpose of the loop is to handle > cases where we have to remove *some* (not all) XIDs from before > cutoff_xid that appear in a

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Peter Geoghegan
On Fri, Feb 25, 2022 at 2:00 PM Peter Geoghegan wrote: > > Hm. I guess I'll have to look at the code for it. It doesn't immediately > > "feel" quite right. > > I kinda think it might be. Please let me know if you see a problem > with what I've said. Oh, wait. I have a better idea of what you

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 14:00:12 -0800, Peter Geoghegan wrote: > On Thu, Feb 24, 2022 at 11:14 PM Andres Freund wrote: > > I am not a fan of the backstop terminology. It's still the reason we need to > > do freezing for correctness reasons. > > Thanks for the review! > > I'm not wedded to that

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-02-25 Thread Peter Geoghegan
On Thu, Feb 24, 2022 at 11:14 PM Andres Freund wrote: > I am not a fan of the backstop terminology. It's still the reason we need to > do freezing for correctness reasons. Thanks for the review! I'm not wedded to that particular terminology, but I think that we need something like it. Open to

Re: why do hash index builds use smgrextend() for new splitpoint pages

2022-02-25 Thread Melanie Plageman
On Thu, Feb 24, 2022 at 10:24 PM Amit Kapila wrote: > > On Fri, Feb 25, 2022 at 4:41 AM Melanie Plageman > wrote: > > > > I'm trying to understand why hash indexes are built primarily in shared > > buffers except when allocating a new splitpoint's worth of bucket pages > > -- which is done with

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 14:25:52 +0800, Julien Rouhaud wrote: > > Basically the type of stats we're trying to move to dynamic shared memory is > > about counters that should persist for a while and are accumulated across > > connections etc. Whereas backend_status.c is more about tracking the current >

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-25 Thread Andres Freund
On 2022-02-25 20:19:24 +, Jacob Champion wrote: > From 2fde60a6bc3739f1894c8c264120e4fa0f04df64 Mon Sep 17 00:00:00 2001 > From: Jacob Champion > Date: Mon, 14 Feb 2022 08:10:53 -0800 > Subject: [PATCH v3] Add API to retrieve authn_id from SQL > The authn_id field in MyProcPort is currently

Re: [PATCH] pg_permissions

2022-02-25 Thread Chapman Flack
I would be happy to review this patch, but a look through the email leaves me thinking it may still be waiting on a C implementation of pg_get_acl(). Is that right? And perhaps a view rename to pg_privileges, following Peter's comment?

Re: Is it correct to update db state in control file as "shutting down" during end-of-recovery checkpoint?

2022-02-25 Thread Nathan Bossart
On Mon, Jan 31, 2022 at 04:25:21PM +0900, Michael Paquier wrote: > At the end of the day, it may be better to just let this stuff be. > Another argument for doing nothing is that this could cause > hard-to-spot conflicts when it comes to back-patch something. This one has been quiet for a while.

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-25 Thread Jacob Champion
On Fri, 2022-02-25 at 16:28 +, Jacob Champion wrote: > Ha, opr_sanity caught my use of cstring. I'll work on a fix later > today. Fixed in v3. --Jacob From 2fde60a6bc3739f1894c8c264120e4fa0f04df64 Mon Sep 17 00:00:00 2001 From: Jacob Champion Date: Mon, 14 Feb 2022 08:10:53 -0800 Subject:

Re: Race conditions in 019_replslot_limit.pl

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 15:07:01 -0500, Tom Lane wrote: > Andres Freund writes: > > Seems to suggest something is holding a problematic lock in a way that I do > > not understand yet: > > > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake=2022-02-23%2013%3A47%3A20=recovery-check >

Re: Race conditions in 019_replslot_limit.pl

2022-02-25 Thread Tom Lane
Andres Freund writes: > Seems to suggest something is holding a problematic lock in a way that I do > not understand yet: > https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake=2022-02-23%2013%3A47%3A20=recovery-check > 2022-02-23 09:09:52.299 EST [2022-02-23 09:09:52 EST

Re: Commitfest manager for 2022-03

2022-02-25 Thread David Steele
Hi Greg, On 2/25/22 12:39, Greg Stark wrote: I would like to volunteer. On Fri, 25 Feb 2022 at 05:31, Julien Rouhaud wrote: Is there any volunteer? Since I've done the March CF for the last seven years, I thought it would be good if I chimed in. I've been agonizing a bit because I have

Re: Add index scan progress to pg_stat_progress_vacuum

2022-02-25 Thread Nathan Bossart
On Wed, Feb 23, 2022 at 10:41:36AM -0800, Peter Geoghegan wrote: > On Wed, Feb 23, 2022 at 10:02 AM Imseih (AWS), Sami > wrote: >> If the failsafe kicks in midway through a vacuum, the number indexes_total >> will not be reset to 0. If INDEX_CLEANUP is turned off, then the value will >> be 0

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 14:10:39 -0500, Tom Lane wrote: > I'm happy to add support for custom auth methods if they can use > a protocol that's safer than cleartext-password. But if that's the > only feasible option, then we're just encouraging people to use > insecure methods. It looks like scram can

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Jacob Champion
On Fri, 2022-02-25 at 13:40 -0500, Jonathan S. Katz wrote: > I'm generally in favor of being able to support additional > authentication methods, the first one coming to mind is supporting OIDC. > Having a pluggable auth infrastructure could possibly make such efforts > easier. I'm definitely

Re: Allow async standbys wait for sync replication (was: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers)

2022-02-25 Thread Nathan Bossart
On Fri, Feb 25, 2022 at 08:31:37PM +0530, Bharath Rupireddy wrote: > Thanks Satya and others for the inputs. Here's the v1 patch that > basically allows async wal senders to wait until the sync standbys > report their flush lsn back to the primary. Please let me know your > thoughts. I haven't

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 16:25:01 -0300, Euler Taveira wrote: > On Fri, Feb 25, 2022, at 11:52 AM, Greg Stark wrote: > > On Tue, 25 Jan 2022 at 01:32, Andres Freund wrote: > > > > > > Hi, > > > > > > I was looking the shared memory stats patch again. > > > > Can you point me to this thread? I looked

Re: Race conditions in 019_replslot_limit.pl

2022-02-25 Thread Andres Freund
Hi, On 2022-02-22 18:06:24 -0800, Andres Freund wrote: > On 2022-02-18 15:14:15 -0800, Andres Freund wrote: > > I'm running out of ideas for how to try to reproduce this. I think we might > > need some additional debugging information to get more information from the > > buildfarm. > > > I'm

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Euler Taveira
On Fri, Feb 25, 2022, at 11:52 AM, Greg Stark wrote: > On Tue, 25 Jan 2022 at 01:32, Andres Freund wrote: > > > > Hi, > > > > I was looking the shared memory stats patch again. > > Can you point me to this thread? I looked for it but couldn't find it.

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Tom Lane
Jeff Davis writes: > On Fri, 2022-02-25 at 12:39 -0500, Tom Lane wrote: >> My point is that sending cleartext passwords over the wire is an >> insecure-by-definition protocol that we shouldn't be encouraging >> more use of. > We can require custom auth entries in pg_hba.conf to also specify >

Re: Fix overflow in justify_interval related functions

2022-02-25 Thread Nathan Bossart
On Fri, Feb 25, 2022 at 10:30:57AM -0500, Joseph Koshakow wrote: > Just checking because I'm not very familiar with the process, > are there any outstanding items that I need to do for this patch? Unless someone has additional feedback, I don't think so. -- Nathan Bossart Amazon Web Services:

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-25 Thread David Christensen
Bharath Rupireddy writes: > On Fri, Feb 25, 2022 at 12:36 AM David Christensen > wrote: >> >> Greetings, >> >> This patch adds the ability to specify a RelFileNode and optional BlockNum >> to limit output of >> pg_waldump records to only those which match the given criteria. This >> should

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Jeff Davis
On Fri, 2022-02-25 at 12:39 -0500, Tom Lane wrote: > My point is that sending cleartext passwords over the wire is an > insecure-by-definition protocol that we shouldn't be encouraging > more use of. We can require custom auth entries in pg_hba.conf to also specify local, hostssl or hostgssenc.

Re: Two noncritical bugs of pg_waldump

2022-02-25 Thread Andres Freund
Hi, On 2022-02-14 18:18:47 +0900, Kyotaro Horiguchi wrote: > > If I give an empty file to the tool it complains as the follows. > > > > > pg_waldump: fatal: could not read file "hoge": No such file or directory > > > > No, the file exists. The cause is it reads uninitialized errno to > >

Re: Using operators to do query hints

2022-02-25 Thread Bruce Momjian
On Tue, Feb 22, 2022 at 04:12:15PM -0500, Greg Stark wrote: > I've been playing with an idea I had a while back. Basically that it > would be useful to have some "noop" operators that are used purely to > influence the planner. > > For context I've suggested in the past that there are two

Re: C++ Trigger Framework

2022-02-25 Thread Bruce Momjian
On Tue, Feb 22, 2022 at 11:33:05PM +0200, Shmuel Kamensky wrote: > Hi Nathan, > > Thanks for the reply. I did indeed read that document and it's a great place > to > get started. But it doesn't quite answer my questions. Do you personally have > experience writing software in C++ that interacts

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Jonathan S. Katz
On 2/25/22 12:39 PM, Tom Lane wrote: Jeff Davis writes: On Thu, 2022-02-24 at 20:47 -0500, Tom Lane wrote: ... and, since we can't readily enforce that the client only sends those cleartext passwords over suitably-encrypted connections, this could easily be a net negative for security. Not

Re: trigger example for plsample

2022-02-25 Thread Chapman Flack
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested This patch is straightforward, does what it says, and passes the

Re: Commitfest manager for 2022-03

2022-02-25 Thread Greg Stark
I would like to volunteer. On Fri, 25 Feb 2022 at 05:31, Julien Rouhaud wrote: > > Hi, > > The final commitfest for pg15 will start in a few days, and I didn't see any > discussion on it or anyone volunteering to be a CFM. > > I thought it would be a good idea to send this reminder now and avoid

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Julien Rouhaud
On Sat, Feb 26, 2022 at 02:30:36AM +0800, Julien Rouhaud wrote: > On Fri, Feb 25, 2022 at 06:49:42PM +0100, Matthias van de Meent wrote: > > > > The point I was trying to make was "If cps->ckpt_flags is > > CHECKPOINT_IMMEDIATE, we hurry up to start the new checkpoint that is > > actually

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Julien Rouhaud
On Fri, Feb 25, 2022 at 06:49:42PM +0100, Matthias van de Meent wrote: > > The point I was trying to make was "If cps->ckpt_flags is > CHECKPOINT_IMMEDIATE, we hurry up to start the new checkpoint that is > actually immediate". That doesn't mean that this checkpoint was > created with IMMEDIATE or

Re: Some optimisations for numeric division

2022-02-25 Thread Tom Lane
Dean Rasheed writes: > And another update following feedback from the cfbot. This patchset LGTM. On my machine there doesn't seem to be any measurable performance change for the numeric regression test, but numeric_big gets about 15% faster. The only additional thought I have, based on your

Re: [PATCH] Add reloption for views to enable RLS

2022-02-25 Thread Dean Rasheed
On Fri, 18 Feb 2022 at 14:57, Laurenz Albe wrote: > > Here is a new version, with improved documentation and the option renamed > to "check_permissions_owner". I just prefer the shorter form. > Re-reading this thread, I think I preferred the name "security_invoker". The main objection seemed to

Re: convert libpq uri-regress tests to tap test

2022-02-25 Thread Andres Freund
Hi, On 2022-02-24 08:46:23 -0800, Andres Freund wrote: > I'm mildly inclined to only do 0001 and 0002 for now. We'd not loose msvc > coverage, because it already doesn't build the test. Once we've ironed that > stuff out, we could do 0003? >From what I can see in the buildfarm client, we'd not

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Matthias van de Meent
On Fri, 25 Feb 2022 at 17:35, Julien Rouhaud wrote: > > On Fri, Feb 25, 2022 at 08:53:50PM +0530, Nitin Jadhav wrote: > > > > > > I'm not sure what Matthias meant, but as far as I know there's no > > > fundamental > > > difference between checkpoint with and without the CHECKPOINT_IMMEDIATE > >

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 09:33:45 -0800, Jeff Davis wrote: > On Thu, 2022-02-24 at 20:47 -0500, Tom Lane wrote: > > ... and, since we can't readily enforce that the client only sends > > those cleartext passwords over suitably-encrypted connections, this > > could easily be a net negative for security.

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Tom Lane
Jeff Davis writes: > On Thu, 2022-02-24 at 20:47 -0500, Tom Lane wrote: >> ... and, since we can't readily enforce that the client only sends >> those cleartext passwords over suitably-encrypted connections, this >> could easily be a net negative for security. Not sure that I think >> it's a

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Jeff Davis
On Thu, 2022-02-24 at 20:47 -0500, Tom Lane wrote: > ... and, since we can't readily enforce that the client only sends > those cleartext passwords over suitably-encrypted connections, this > could easily be a net negative for security. Not sure that I think > it's a good idea. I don't

Re: Proposal: Support custom authentication methods using hooks

2022-02-25 Thread Jeff Davis
On Thu, 2022-02-24 at 19:47 -0800, Andres Freund wrote: > Why is it restricted to that? You could do sasl negotiation as well > from what > I can see? And that'd theoretically also allow to negotiate whether > the client > supports different ways of doing auth? Not saying that that's easy, > but

Re: Readd use of TAP subtests

2022-02-25 Thread Andrew Dunstan
On 2/25/22 11:41, Andres Freund wrote: > Hi, > > On 2022-02-25 09:43:20 -0500, Andrew Dunstan wrote: >> AIUI TAP consumers are supposed to ignore lines they don't understand. > Are they? > > In http://testanything.org/tap-version-13-specification.html there's: > > "Lines written to standard

Re: should vacuum's first heap pass be read-only?

2022-02-25 Thread Peter Geoghegan
On Fri, Feb 25, 2022 at 5:06 AM Dilip Kumar wrote: > Based on this discussion, IIUC, we are saying that now we will do the > lazy_scan_heap every time like we are doing now. And we will > conditionally skip the index vacuum for all or some of the indexes and > then based on how much index vacuum

Re: BufferAlloc: don't take two simultaneous locks

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 12:51:22 +0300, Yura Sokolov wrote: > > > + * The usage_count starts out at 1 so that the buffer can survive one > > > + * clock-sweep pass. > > > + * > > > + * We use direct atomic OR instead of Lock+Unlock since no other backend > > > + * could be interested in the

Re: Add parameter jit_warn_above_fraction

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 17:28:41 +0100, Magnus Hagander wrote: > On Fri, Feb 25, 2022 at 5:20 PM Andres Freund wrote: > > On 2022-02-25 16:16:01 +0100, Magnus Hagander wrote: > > > This patch adds a configuration parameter jit_warn_above_fraction that > > > will cause a warning to be logged if the

Mingw task for Cirrus CI

2022-02-25 Thread Melih Mutlu
Hi All, I've been working on adding Windows+MinGW environment into cirrus-ci tasks (discussion about ci is here [1]). It uses MSYS2 to set the environment. UCRT is chosen as C standard library, instead of MSVCRT. The task configures postgres with features that are available in MSYS2 (see

Re: Expose JIT counters/timing in pg_stat_statements

2022-02-25 Thread Julien Rouhaud
On Fri, Feb 25, 2022 at 05:38:45PM +0100, Magnus Hagander wrote: > > Per some off-list discussion with Julien, we have clearly been talking > in slightly different terms. So let's summarize the options into what > theÿ́d actually be: > > Option 0: what is int he patch now > > Option 1: > jit_count

Re: Readd use of TAP subtests

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 09:43:20 -0500, Andrew Dunstan wrote: > AIUI TAP consumers are supposed to ignore lines they don't understand. Are they? In http://testanything.org/tap-version-13-specification.html there's: "Lines written to standard output matching /^(not )?ok\b/ must be interpreted as test

Re: Expose JIT counters/timing in pg_stat_statements

2022-02-25 Thread Dmitry Dolgov
> On Fri, Feb 25, 2022 at 04:19:27PM +0100, Magnus Hagander wrote: > On Fri, Feb 25, 2022 at 2:33 PM Julien Rouhaud wrote: > > > > Hi, > > > > On Fri, Feb 25, 2022 at 02:06:29PM +0100, Magnus Hagander wrote: > > > Here's a patch to add the sum of timings for JIT counters to > > >

Re: Expose JIT counters/timing in pg_stat_statements

2022-02-25 Thread Magnus Hagander
On Fri, Feb 25, 2022 at 4:41 PM Magnus Hagander wrote: > > On Fri, Feb 25, 2022 at 4:40 PM Julien Rouhaud wrote: > > > > On Fri, Feb 25, 2022 at 04:19:27PM +0100, Magnus Hagander wrote: > > > > > > So just to be clear, you're basically thinking: > > > > > > jit_count = count of entries where

Re: Readd use of TAP subtests

2022-02-25 Thread Jacob Champion
On Fri, 2022-02-25 at 16:35 +, Jacob Champion wrote: > On Fri, 2022-02-25 at 09:43 -0500, Andrew Dunstan wrote: > > AIUI TAP consumers are supposed to ignore lines they don't understand. > > It's undefined behavior [1]: And of course the minute I send this I notice that I've linked the v13

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Julien Rouhaud
On Fri, Feb 25, 2022 at 08:53:50PM +0530, Nitin Jadhav wrote: > > > > I'm not sure what Matthias meant, but as far as I know there's no > > fundamental > > difference between checkpoint with and without the CHECKPOINT_IMMEDIATE > > flag, > > and there's also no scheduling for multiple

Re: Readd use of TAP subtests

2022-02-25 Thread Jacob Champion
On Fri, 2022-02-25 at 09:43 -0500, Andrew Dunstan wrote: > AIUI TAP consumers are supposed to ignore lines they don't understand. It's undefined behavior [1]: > Any output that is not a version, a plan, a test line, a YAML block, > a diagnostic or a bail out is incorrect. How a harness handles

Re: Add parameter jit_warn_above_fraction

2022-02-25 Thread Magnus Hagander
On Fri, Feb 25, 2022 at 5:20 PM Andres Freund wrote: > > Hi, > > On 2022-02-25 16:16:01 +0100, Magnus Hagander wrote: > > This patch adds a configuration parameter jit_warn_above_fraction that > > will cause a warning to be logged if the fraction of time spent on > > doing JIT is bigger than the

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-02-25 Thread Jacob Champion
On Thu, 2022-02-24 at 20:44 +, Jacob Champion wrote: > That simplifies things. PFA a smaller v2; if pgaudit can't make use of > libpq-be.h for some reason, then I guess we can tailor a fix to that > use case. Ha, opr_sanity caught my use of cstring. I'll work on a fix later today. --Jacob

Re: Readd use of TAP subtests

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 14:39:15 +0100, Peter Eisentraut wrote: > On 24.02.22 16:00, Andres Freund wrote: > > I've incidentally played with subtests yesterdays, when porting > > src/interfaces/libpq/test/regress.pl to a tap test. Unfortunately it seems > > that subtests aren't actually specified in the

Re: Add parameter jit_warn_above_fraction

2022-02-25 Thread Justin Pryzby
On Fri, Feb 25, 2022 at 04:16:01PM +0100, Magnus Hagander wrote: > + { > + {"jit_warn_above_fraction", PGC_SUSET, LOGGING_WHEN, > + gettext_noop("Sets the fraction of query time spent on > JIT before writing" > + "a

Re: Add parameter jit_warn_above_fraction

2022-02-25 Thread Andres Freund
Hi, On 2022-02-25 16:16:01 +0100, Magnus Hagander wrote: > This patch adds a configuration parameter jit_warn_above_fraction that > will cause a warning to be logged if the fraction of time spent on > doing JIT is bigger than the specified one. For example, this can be > used to track down those

Re: Add parameter jit_warn_above_fraction

2022-02-25 Thread Julien Rouhaud
Hi, On Fri, Feb 25, 2022 at 04:16:01PM +0100, Magnus Hagander wrote: > This patch adds a configuration parameter jit_warn_above_fraction that > will cause a warning to be logged if the fraction of time spent on > doing JIT is bigger than the specified one. For example, this can be > used to track

Re: Expose JIT counters/timing in pg_stat_statements

2022-02-25 Thread Magnus Hagander
On Fri, Feb 25, 2022 at 4:40 PM Julien Rouhaud wrote: > > On Fri, Feb 25, 2022 at 04:19:27PM +0100, Magnus Hagander wrote: > > > > So just to be clear, you're basically thinking: > > > > jit_count = count of entries where jit_functions>0 > > jit_functions = > > jit_optimizatinos = count of

Re: Expose JIT counters/timing in pg_stat_statements

2022-02-25 Thread Julien Rouhaud
On Fri, Feb 25, 2022 at 04:19:27PM +0100, Magnus Hagander wrote: > > So just to be clear, you're basically thinking: > > jit_count = count of entries where jit_functions>0 > jit_functions = > jit_optimizatinos = count of entries where time spent on jit_optimizations > 0 > > etc? Yes exactly,

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-25 Thread David Christensen
On Fri, Feb 25, 2022 at 7:08 AM Japin Li wrote: > > On Fri, 25 Feb 2022 at 20:48, David Christensen < > david.christen...@crunchydata.com> wrote: > >> Cool. I think we can report an error instead of reading wal files, > >> if the tablespace, database, or relation is invalid. Does there any >

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-02-25 Thread David Christensen
On Fri, Feb 25, 2022 at 7:33 AM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > Thanks for the patch. This is not adding something that users can't do > right now, but definitely improves the usability of the pg_waldump as > it avoids external filterings. Also, it can give

Re: Fix overflow in justify_interval related functions

2022-02-25 Thread Joseph Koshakow
Just checking because I'm not very familiar with the process, are there any outstanding items that I need to do for this patch? - Joe Koshakow

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Nitin Jadhav
> > Thank you Alvaro and Matthias for your views. I understand your point > > of not updating the progress-report flag here as it just checks > > whether the CHECKPOINT_IMMEDIATE is set or not and takes an action > > based on that but it doesn't change the checkpoint flags. I will > > modify the

Re: Expose JIT counters/timing in pg_stat_statements

2022-02-25 Thread Magnus Hagander
On Fri, Feb 25, 2022 at 2:33 PM Julien Rouhaud wrote: > > Hi, > > On Fri, Feb 25, 2022 at 02:06:29PM +0100, Magnus Hagander wrote: > > Here's a patch to add the sum of timings for JIT counters to > > pg_stat_statements, as a way to follow-up on if JIT is doing a good or > > a bad job in a

Re: Size functions inconsistent results

2022-02-25 Thread Fabrízio de Royes Mello
On Fri, Feb 25, 2022 at 12:10 PM Japin Li wrote: > > > I think, you forget the index size of toast table. > > with relations as ( > select schemaname, relname, relid > from pg_stat_user_tables > where relname = 'test_size' > ), > sizes as ( > select > schemaname, > r.relname, > >

Add parameter jit_warn_above_fraction

2022-02-25 Thread Magnus Hagander
This patch adds a configuration parameter jit_warn_above_fraction that will cause a warning to be logged if the fraction of time spent on doing JIT is bigger than the specified one. For example, this can be used to track down those cases where JIT ends up taking 90% of the query runtime because of

Re: Size functions inconsistent results

2022-02-25 Thread Japin Li
On Fri, 25 Feb 2022 at 22:58, Fabrízio de Royes Mello wrote: > Hi all, > > While doing some work using our functions [1] for calculate relations size > I noticed an inconsistency between pg_total_relation_size and calculate > everything separately, have a look in this example: > > fabrizio=#

Fix typo in logicalfuncs.c - :%s/private date/Private data

2022-02-25 Thread Bharath Rupireddy
Hi, Here's a tiny patch to do $subject. Regards, Bharath Rupireddy. v1-0001-Fix-typo-in-logicalfuncs.c.patch Description: Binary data

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Nitin Jadhav
> + if ((ckpt_flags & > +(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_END_OF_RECOVERY)) == 0) > + { > > This code (present at multiple places) looks a little ugly to me, what > we can do instead is add a macro probably named IsShutdownCheckpoint() > which does the above check

Allow async standbys wait for sync replication (was: Disallow quorum uncommitted (with synchronous standbys) txns in logical replication subscribers)

2022-02-25 Thread Bharath Rupireddy
On Thu, Jan 6, 2022 at 1:29 PM SATYANARAYANA NARLAPURAM wrote: > > Consider a cluster formation where we have a Primary(P), Sync Replica(S1), > and multiple async replicas for disaster recovery and read scaling (within > the region and outside the region). In this setup, S1 is the preferred >

Size functions inconsistent results

2022-02-25 Thread Fabrízio de Royes Mello
Hi all, While doing some work using our functions [1] for calculate relations size I noticed an inconsistency between pg_total_relation_size and calculate everything separately, have a look in this example: fabrizio=# create table test_size (id bigserial primary key, toast_column text); CREATE

Re: Report checkpoint progress with pg_stat_progress_checkpoint (was: Report checkpoint progress in server logs)

2022-02-25 Thread Nitin Jadhav
> Thank you Alvaro and Matthias for your views. I understand your point > of not updating the progress-report flag here as it just checks > whether the CHECKPOINT_IMMEDIATE is set or not and takes an action > based on that but it doesn't change the checkpoint flags. I will > modify the code but I

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Greg Stark
On Tue, 25 Jan 2022 at 01:32, Andres Freund wrote: > > Hi, > > I was looking the shared memory stats patch again. Can you point me to this thread? I looked for it but couldn't find it. -- greg

Re: Readd use of TAP subtests

2022-02-25 Thread Andrew Dunstan
On 2/25/22 08:39, Peter Eisentraut wrote: > On 24.02.22 16:00, Andres Freund wrote: >> I've incidentally played with subtests yesterdays, when porting >> src/interfaces/libpq/test/regress.pl to a tap test. Unfortunately it >> seems >> that subtests aren't actually specified in the tap format,

Re: set TESTDIR from perl rather than Makefile

2022-02-25 Thread Andrew Dunstan
On 2/24/22 20:17, Justin Pryzby wrote: > On Mon, Feb 21, 2022 at 07:00:54AM -0500, Andrew Dunstan wrote: >> On 2/19/22 18:53, Justin Pryzby wrote: >>> On Sat, Feb 19, 2022 at 05:41:49PM -0600, Justin Pryzby wrote: I rebased and fixed the check-guc script to work, made it work with vpath

Re: Readd use of TAP subtests

2022-02-25 Thread Peter Eisentraut
On 24.02.22 16:00, Andres Freund wrote: I've incidentally played with subtests yesterdays, when porting src/interfaces/libpq/test/regress.pl to a tap test. Unfortunately it seems that subtests aren't actually specified in the tap format, and that different libraries generate different output

  1   2   >