Re: Support for NSS as a libpq TLS backend

2021-03-24 Thread Stephen Frost
Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > On Wed, 2021-03-24 at 13:00 -0400, Stephen Frost wrote: > > * Jacob Champion (pchamp...@vmware.com) wrote: > > > Right, but to clarify -- I was asking if *NSS* supports loading and > > > using separate ce

Re: Change default of checkpoint_completion_target

2021-03-24 Thread Stephen Frost
Greetings, * Bossart, Nathan (bossa...@amazon.com) wrote: > On 3/23/21, 12:19 PM, "Stephen Frost" wrote: > > * Bossart, Nathan (bossa...@amazon.com) wrote: > > > LGTM. I just have a few small wording suggestions. > > > > Agreed, those looked like g

Re: Support for NSS as a libpq TLS backend

2021-03-24 Thread Stephen Frost
Greetings Jacob, * Jacob Champion (pchamp...@vmware.com) wrote: > On Wed, 2021-03-24 at 09:28 +0900, Michael Paquier wrote: > > On Wed, Mar 24, 2021 at 12:05:35AM +, Jacob Champion wrote: > > > I can work around it temporarily for the > > > tests, but this will be a problem if any libpq

Re: Autovacuum worker doesn't immediately exit on postmaster death

2021-03-24 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Mon, Mar 22, 2021 at 04:07:12PM -0400, Robert Haas wrote: > > On Mon, Mar 22, 2021 at 1:48 PM Stephen Frost wrote: > >> Thanks for that. Attached is just a rebased version with a commit > >> message added.

Re: Disable WAL logging to speed up data loading

2021-03-24 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: Stephen Frost > > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > As for data loading tools, surely they support loading data into UNLOGGED > > tables and it's certa

Re: Change default of checkpoint_completion_target

2021-03-23 Thread Stephen Frost
:00 2001 From: Stephen Frost Date: Tue, 19 Jan 2021 13:53:34 -0500 Subject: [PATCH] Change checkpoint_completion_target default to 0.9 Common recommendations are that the checkpoint should be spread out as much as possible, provided we avoid having it take too long. This change updates the defa

Re: Support for NSS as a libpq TLS backend

2021-03-23 Thread Stephen Frost
Greetings, * Daniel Gustafsson (dan...@yesql.se) wrote: > > On 22 Mar 2021, at 00:49, Stephen Frost wrote: > > Thanks for the review! Below is a partial response, I haven't had time to > address all your review comments yet but I wanted to submit a rebased patchset > directly

Re: Disable WAL logging to speed up data loading

2021-03-23 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: Stephen Frost > > First- what are you expecting would actually happen during crash recovery in > > this specific case with your proposed new WAL level? > ... > > I'm not suggesting it's

Re: Change default of checkpoint_completion_target

2021-03-23 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Mon, Mar 22, 2021 at 01:11:00PM -0400, Stephen Frost wrote: > > Unless there's anything further on this, I'll plan to commit it tomorrow > > or Wednesday. > > Cool, looks fine to me. > > This version

Re: Autovacuum worker doesn't immediately exit on postmaster death

2021-03-22 Thread Stephen Frost
Greetings, * Thomas Munro (thomas.mu...@gmail.com) wrote: > On Fri, Dec 11, 2020 at 7:57 AM Stephen Frost wrote: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > > The if-we're-going-to-delay-anyway path in vacuum_delay_point seems > > > OK to add a touch more overhead

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Mon, 2021-03-22 at 11:05 -0400, Stephen Frost wrote: > > > Perhaps allowing to set unlogged tables to logged ones without writing WAL > > > is a more elegant way to do that, but I cannot see how that would be

Re: Change default of checkpoint_completion_target

2021-03-22 Thread Stephen Frost
this > patch follows the current convention. Agreed. Unless there's anything further on this, I'll plan to commit it tomorrow or Wednesday. Thanks! Stephen From 3ebe08dee4b9dfe2dff51fd1bad2eb36834e82ed Mon Sep 17 00:00:00 2001 From: Stephen Frost Date: Tue, 19 Jan 2021 13:53:34 -0500

Re: Add docs stub for recovery.conf

2021-03-22 Thread Stephen Frost
nything really changed). Unless someone wants to speak up, I'll commit this soonish (hopefully tomorrow, but at least sometime later this week). Thanks! Stephen From 000cd577d6dbb9d6d6c571e2302657f1252e6a56 Mon Sep 17 00:00:00 2001 From: Stephen Frost Date: Mon, 22 Mar 2021 12:45:41 -0400 S

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Mon, 2021-03-22 at 09:46 -0400, Stephen Frost wrote: > > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > > From: Stephen Frost > > > > The argument here seems to stem from the

Re: Disable WAL logging to speed up data loading

2021-03-22 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: Stephen Frost > > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > > inside the transaction before starting the 'COPY' command is 'too hard'. > > > I co

Re: Support for NSS as a libpq TLS backend

2021-03-21 Thread Stephen Frost
Greetings, * Daniel Gustafsson (dan...@yesql.se) wrote: > Attached is a rebase which attempts to fix the cfbot Appveyor failure, there > were missing HAVE_ defines for MSVC. > Subject: [PATCH v30 1/9] nss: Support libnss as TLS library in libpq > > This commit contains the frontend and backend

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-21 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > If I understand what you are proposing, all stats views would become > completely volatile, without even within-query consistency. That really > is not gonna work. As an example, you could get not-even-self-consistent > results from a join to

Re: recovery_init_sync_method=wal

2021-03-21 Thread Stephen Frost
Greetings, * Thomas Munro (thomas.mu...@gmail.com) wrote: > On Mon, Mar 22, 2021 at 4:31 AM Stephen Frost wrote: > > Presuming that we do add to the documentation the language to document > > what's assumed (and already done by modern backup tools) that they're > > fsync'i

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-21 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > I am working on Kyotaro Horiguchi's shared memory stats patch [1] with > the goal of getting it into a shape that I'd be happy to commit. That > thread is quite long and most are probably skipping over new messages in > it. Awesome, +1.

Re: recovery_init_sync_method=wal

2021-03-21 Thread Stephen Frost
Greetings, * Thomas Munro (thomas.mu...@gmail.com) wrote: > 2. You made a file system-level copy of a cluster that you shut down > cleanly first, using cp, tar, scp, rsync, xmodem etc. Now you start > up the copy. Its checkpoint is a forgery. (Maybe our manual should > mention this problem

Re: Log message for GSS connection is missing once connection authorization is successful.

2021-03-21 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Sat, Mar 20, 2021 at 05:37:47PM +0900, Michael Paquier wrote: > > It seems to me that this would make the tests faster, that the test > > would not need to wait for the logging collector and that the code > > could just use

Re: Disable WAL logging to speed up data loading

2021-03-19 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: David Steele > > After reading through the thread (but not reading the patch) I am -1 on > > this proposal. > > > > The feature seems ripe for abuse and misunderstanding, and as has been > > noted in the

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > On 2021-Mar-18, Stephen Frost wrote: > > > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > > > it "rela

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > it "relations that are not WAL-logged"? I suppose we already have a > term for this; I'm not sure it's a good idea to invent a different term > that is only

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote: > > I have made significant progress on the cluster file encryption feature so > > it is time for me to post a new set of patches. > > Here is a rebase, to keep the cfbot green.

Re: WIP: WAL prefetch (another approach)

2021-03-17 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > Right, I was just going to point out the FPIs are not necessary - what > matters is the presence of long streaks of WAL records touching the same > set of blocks. But people with workloads where this is common likely > don't need

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-16 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > > >> I think e.g. prefetch_targblock could be moved to the next #ifdef, which > > >> will eliminate the one-line ifdef. > > > > > > Sure, done in the attached. > > > > > > Than

Re: Enhance traceability of wal_level changes for backup management

2021-03-15 Thread Stephen Frost
Greetings, * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > From: David Steele > > As a backup software author, I don't see this feature as very useful. > > > > The problem is that there are lots of ways for WAL to go missing so > > monitoring the WAL archive for gaps is

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-11 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 3/12/21 1:11 AM, Stephen Frost wrote: > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > >> On 3/8/21 8:42 PM, Stephen Frost wrote: > >>> * Tomas Vondra (tomas.von...@enterprisedb.com) wro

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-11 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 3/8/21 8:42 PM, Stephen Frost wrote: > > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > >> On 2/10/21 11:10 PM, Stephen Frost wrote: > >>> * Heikki Linnakangas (hlinn...@iki.fi) wro

Re: automatic analyze: readahead - add "IO read time" log message

2021-03-08 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 2/10/21 11:10 PM, Stephen Frost wrote: > > * Heikki Linnakangas (hlinn...@iki.fi) wrote: > >> On 05/02/2021 23:22, Stephen Frost wrote: > >>> Unless there's anything else on this, I'll commit

Re: Interest in GSoC 2021 Projects

2021-02-26 Thread Stephen Frost
* and the > potential mentor is *Andreas Scherbaum* and *Stephen Frost*. > > I would like to learn more about these two projects but failed to contact > the mentors. How can I contact them? Also, I really hope to join the > project. Are there any suggestions on application? Gla

Re: SSL SNI

2021-02-26 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@enterprisedb.com) wrote: > A customer asked about including Server Name Indication (SNI) into the SSL > connection from the client, so they can use an SSL-aware proxy to route > connections. There was a thread a few years ago where this was briefly

Re: Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Stephen Frost
Greetings, * Daniele Varrazzo (daniele.varra...@gmail.com) wrote: > I am designing and implementing a connection pool for psycopg3 [1][2]. > Some of the inspiration is coming from HikariCP [3], a Java connection > pool. > > One of the HikariCP configuration parameters is "maxLifetime", whose >

Re: WIP: WAL prefetch (another approach)

2021-02-14 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 2/13/21 10:39 PM, Stephen Frost wrote: > >* Andres Freund (and...@anarazel.de) wrote: > >>On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote: > >>>Yeah, that's a good point. I think it'd make

Re: WIP: WAL prefetch (another approach)

2021-02-13 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote: > > Yeah, that's a good point. I think it'd make sense to keep track of recent > > FPIs and skip prefetching such blocks. But how exactly should we implement > > that, how many blocks do we

Re: automatic analyze: readahead - add "IO read time" log message

2021-02-10 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > On 05/02/2021 23:22, Stephen Frost wrote: > >Unless there's anything else on this, I'll commit these sometime next > >week. > > One more thing: Instead of using 'effective_io_concurrency' GUC di

Re: WIP: WAL prefetch (another approach)

2021-02-10 Thread Stephen Frost
Greetings, * Thomas Munro (thomas.mu...@gmail.com) wrote: > Rebase attached. > Subject: [PATCH v15 4/6] Prefetch referenced blocks during recovery. > diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml > index 4b60382778..ac27392053 100644 > --- a/doc/src/sgml/config.sgml > +++

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote: > > > I looked further. First, I don't think we are going to be able to > > > protect at all against users who have _write_ access on the OS running >

Re: automatic analyze: readahead - add "IO read time" log message

2021-02-05 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > On 13/01/2021 23:17, Stephen Frost wrote: > >Would be great to get a review / comments from others as to if there's > >any concerns. I'll admit that it seems reasonably straight-forward to > >me, but hey, I wrote

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > On 05/02/2021 21:16, Andrew Dunstan wrote: > >On 2/5/21 10:54 AM, Stephen Frost wrote: > >>* Heikki Linnakangas (hlinn...@iki.fi) wrote: > >>>I ran it for about 2 h on my laptop with the patch I was working o

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote: > > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > > I doubt anyone would actually stipulate that they *guarantee* detection > > >

Re: Fuzz testing COPY FROM parsing

2021-02-05 Thread Stephen Frost
Greetings, * Heikki Linnakangas (hlinn...@iki.fi) wrote: > I've been mucking around with COPY FROM lately, and to test it, I wrote some > tools to generate input files and load them with COPY FROM: > > https://github.com/hlinnaka/pgcopyfuzz Neat! > I used a fuzz testing tool called honggfuzz

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-05 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > On 2021-Feb-05, Mark Rofail wrote: > > I disagree -- I think we should get the second patch in, and consider it > > > a requisite for the other one. > > > > I just want to make sure I got your last message right. We should work on >

Re: Next Commitfest Manager.

2021-02-03 Thread Stephen Frost
Greetings, * Ibrar Ahmed (ibrar.ah...@gmail.com) wrote: > Anyone else already volunteers that? It is my first time so need some > access, if all agree. Thanks for volunteering! That said, our last commitfest tends to be the most difficult as it's the last opportunity for features to land in

Re: Key management with tests

2021-02-03 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Feb 1, 2021 at 07:47:57PM -0500, Bruce Momjian wrote: > > On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > The purpose of cluster

Re: Proposal: Save user's original authenticated identity for logging

2021-02-01 Thread Stephen Frost
Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > On Mon, 2021-02-01 at 18:01 -0500, Stephen Frost wrote: > > Ok.. but what's 'go' mean here? We already have views and such for GSS > > and SSL, is the idea to add another view for LDAP and add in columns > &

Re: Key management with tests

2021-02-01 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote: > > I propose that we meet to discuss what approach we want to use to move TDE > > forward.  We then start a new thread with a proposal on the approach > > and finalize it via

Re: Key management with tests

2021-02-01 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 29, 2021 at 05:40:37PM -0500, Stephen Frost wrote: > > I hope it's pretty clear that I'm also very much in support of both this > > effort with the KMS and of TDE in general- TDE is specifically, > > Yes, th

Re: Proposal: Save user's original authenticated identity for logging

2021-02-01 Thread Stephen Frost
Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > On Mon, 2021-02-01 at 17:01 -0500, Stephen Frost wrote: > > * Jacob Champion (pchamp...@vmware.com) wrote: > > > And I'm not holding > > > my breath for LDAP servers to start implementing federated identi

Re: Proposal: Save user's original authenticated identity for logging

2021-02-01 Thread Stephen Frost
Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > On Mon, 2021-02-01 at 11:49 -0500, Stephen Frost wrote: > > * Magnus Hagander (mag...@hagander.net) wrote: > > > But yes, I think the enforced cleartext password proxying is at the > > > core of the problem. LD

Re: Proposal: Save user's original authenticated identity for logging

2021-02-01 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> This doesn't sound particularly workable: how would you manage > >> inside-the-database permissions? Kerberos isn't going to know > >>

Re: Proposal: Save user's original authenticated identity for logging

2021-02-01 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Greg Stark writes: > > I wonder if there isn't room to handle this the other way around. To > > configure Postgres to not need a CREATE ROLE for every role but > > delegate the user management to the external authentication service. > > > So

Re: Proposal: Save user's original authenticated identity for logging

2021-02-01 Thread Stephen Frost
Greetings, * Magnus Hagander (mag...@hagander.net) wrote: > On Sat, Jan 30, 2021 at 12:21 AM Jacob Champion wrote: > > > I'm also just generally not thrilled with > > > putting much effort into LDAP as it's a demonstrably insecure > > > authentication mechanism. > > > > Because Postgres has to

Re: Key management with tests

2021-01-29 Thread Stephen Frost
hing > > > please > > > let me know): > > > > > > 1) We need to get the current patch for Key Management reviewed and tested > > > further. > > > > > > I spoke to Bruce just now he will see if can get somebody to do this. > > &

Re: Proposal: Save user's original authenticated identity for logging

2021-01-29 Thread Stephen Frost
Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > First, the context: recently I've been digging into the use of third- > party authentication systems with Postgres. One sticking point is the > need to have a Postgres role corresponding to the third-party user > identity, which becomes

Re: Fixing cache pollution in the Kerberos test suite

2021-01-25 Thread Stephen Frost
Greetings, * Jacob Champion (pchamp...@vmware.com) wrote: > I was running tests with a GSS-enabled stack, and ran into some very > long psql timeouts after running the Kerberos test suite. It turns out > the suite pushes test credentials into the user's global cache, and > these no-longer-useful

Re: proposal - idea - enhancing plpgsql FOREACH for JSON, jsonb and hstore

2021-01-23 Thread Stephen Frost
Greetings, * Pavel Stehule (pavel.steh...@gmail.com) wrote: > jsonb with subscripting support can be used as a dictionary object in > plpgsql. > > Can be nice to have support for iteration over a set of tuples (key, > value). Yes, I agree that this would be useful. > FOREACH fieldvar [ KEY

Re: a verbose option for autovacuum

2021-01-23 Thread Stephen Frost
Greetings, On Fri, Jan 22, 2021 at 2:33 PM Tom Lane wrote: > Tommy Li writes: > > Additionally, is there any interest in exposing more vacuum options to be > > run by autovac? Right now it runs FREEZE and ANALYZE, which leaves the > > VERBOSE, SKIP_LOCKED, INDEX_CLEANUP, and TRUNCATE

GSoC 2021

2021-01-22 Thread Stephen Frost
Greetings -hackers, Google Summer of Code is back for 2021! They have changed some of how GSoC is going to work for this year, for a variety of reasons, so please be sure to read this email and posts linked for the updates if you're interested! Everyone interested in suggesting projects or

Re: Change default of checkpoint_completion_target

2021-01-19 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > Any further comments or thoughts on this one? > > This: > > +total time between checkpoints. The default is 0.9, which spreads the > +checkpoint across the entire checkpoin

Re: Change default of checkpoint_completion_target

2021-01-19 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@enterprisedb.com) wrote: > On 2021-01-13 23:10, Stephen Frost wrote: > >>Yes, I agree, and am involved in that thread as well- currently waiting > >>feedback from others about the proposed approach. > >I've tried to pu

Re: Add docs stub for recovery.conf

2021-01-19 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > * Craig Ringer (craig.rin...@enterprisedb.com) wrote: > > On Thu, 14 Jan 2021 at 03:44, Stephen Frost wrote: > > > Alright, how does this look? The new entries are all under the > > > 'obsolete' section to k

Re: Add docs stub for recovery.conf

2021-01-19 Thread Stephen Frost
Greetings, * Craig Ringer (craig.rin...@enterprisedb.com) wrote: > On Thu, 14 Jan 2021 at 03:44, Stephen Frost wrote: > > Alright, how does this look? The new entries are all under the > > 'obsolete' section to keep it out of the main line, but should work to > > 'fix' the

Re: Change default of checkpoint_completion_target

2021-01-13 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > On 2020-Dec-10, Stephen Frost wrote: > > > * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > > > > On Tue, 2020-12-08 at 17:29 +, Bossart, Nathan

Re: automatic analyze: readahead - add "IO read time" log message

2021-01-13 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > > Thanks. I'll do some testing/benchmarking once my machines are free, in > > a couple days perhaps. But as I said before, I don't expect this to > > behave very d

Re: Add docs stub for recovery.conf

2021-01-13 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Dec 4, 2020 at 02:00:23PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Yes, that is pretty much the same thing I was suggesting, except that > > > each rename has its own

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Well, we have eight unused bits in the IV, so we could just increment > > > that for every hint bi

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > > I think one big question is that, since we are using a streaming cipher, > > > do we care about hint bit changes showing to users? I actually don't

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote: > > > This says: > > > > > > > > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements > > > > > > wal_log_hints will be enabled

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Neil Chen (carpenter.nail...@gmail.com) wrote: > On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost wrote: > > This is an interesting question but ultimately I don't think we should > > be looking at this from the perspective of allowing arbitrary changes to >

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 02:19:22PM -0500, Stephen Frost wrote: > > outputs from the GCM encryption and is what provides the integrity / > > authentication of the encrypted data to be able to detect if it's been > > mod

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote: > > Yes, and it avoids the issue of using a single key for too much, which > > is also a concern. The remaining larger issues are to figure out a > >

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > > Although, another approach and one that I've discussed a bit with Bruce, > > is to have more keys- such as a key for temporary files, and perhaps > >

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 08:12:00PM +0900, Masahiko Sawada wrote: > > Looking at the patch, it supports three algorithms but only > > PG_CIPHER_AES_KWP is used in the core for now: > > > > +/* > > + * Supported symmetric encryption algorithm.

Re: Enhance traceability of wal_level changes for backup management

2021-01-08 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > On Thursday, January 7, 2021 2:40 AM Stephen Frost wrote: > > * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > > > You said > > > > The use case I imagined is tha

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 8, 2021 at 03:33:44PM -0500, Stephen Frost wrote: > > > Anyway, I think we need to figure out how to trim. The first part would > > > be to figure out whether we need 128 _and_ 256-bit tests, and then

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings Bruce, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote: > > On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote: > > > I have completed the key management patch with tests created by Stephen > >

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jan 7, 2021 at 04:08:49PM -0300, Álvaro Herrera wrote: > > On 2021-Jan-07, Bruce Momjian wrote: > > > > > All the tests pass now. The current src/test directory is 19MB, and > > > adding these tests takes it to 23MB, or a 20%

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-07 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Thu, Jan 7, 2021 at 1:14 PM Stephen Frost wrote: > > Much of this line of discussion seems to be, incorrectly, focused on my > > mere mention of viewing the use of fsync and checksums as mechanism for > > add

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-07 Thread Stephen Frost
Greetings, * Michael Banck (michael.ba...@credativ.de) wrote: > Am Mittwoch, den 06.01.2021, 13:08 -0800 schrieb Peter Geoghegan: > > On Wed, Jan 6, 2021 at 1:04 PM Michael Banck > > wrote: > > > At least data_checksums=on for Azure Managed Postgres, Amazon RDS and > > > Google Cloud SQL

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-07 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Wed, Jan 6, 2021 at 12:30 PM Stephen Frost wrote: > > As already mentioned, it's also, at least today, far > > simpler to disable checksums than to enable them, which is something > > else to consider when thinking a

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Wed, Jan 6, 2021 at 12:03 PM Stephen Frost wrote: > > Do you really believe it to be wrong? Do we stop performing the correct > > write calls in the correct order to the kernel with fsync being off? If > > the kernel

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Peter Geoghegan (p...@bowt.ie) wrote: > On Wed, Jan 6, 2021 at 11:44 AM Stephen Frost wrote: > > Having fsync off won't actually cause corruption unless you have an OS > > crash or don't sync the disks when you reboot the system though- so it's > > a hedge ag

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-06 13:01:59 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > > imv, enabling page checksums is akin to having fsync enabled by default. > > > > Does it impact perf

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-06 18:27:48 +0100, Magnus Hagander wrote: > > The other argument is that admins can cheaply and quickly turn off > > checksums if they don't want them. > > > > The same cannot be said for turning them *on* again, that's a very >

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-06 12:02:40 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > On 2021-01-04 19:11:43 +0100, Michael Banck wrote: > > > > Am Samstag, den 02.01.2021, 10:47 -0500 schr

Re: Enhance traceability of wal_level changes for backup management

2021-01-06 Thread Stephen Frost
Greetings, * osumi.takami...@fujitsu.com (osumi.takami...@fujitsu.com) wrote: > You said > > The use case I imagined is that the user temporarily > > changes wal_level to 'none' from 'replica' or 'logical' to speed up loading > > and > > changes back to the normal. In this case, the backups

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Jan 6, 2021 at 12:02:40PM -0500, Stephen Frost wrote: > > > It unfortunately also hurts other workloads. If we moved towards a saner > > > compression algorithm that'd perhaps not be an issue anymor

Re: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-06 Thread Stephen Frost
Greetings, * Andres Freund (and...@anarazel.de) wrote: > On 2021-01-04 19:11:43 +0100, Michael Banck wrote: > > Am Samstag, den 02.01.2021, 10:47 -0500 schrieb Stephen Frost: > > > * Michael Paquier (mich...@paquier.xyz) wrote: > > > > On Fri, Jan 01, 2021 at 08:34:3

Re: Safety/validity of resetting permissions by updating system tables

2021-01-05 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Andrew Dunstan writes: > > I think there is probably a good case for some sort of "from scratch" > > option on GRANT. > > Maybe my head's not screwed on straight this morning, but it seems > to me that any such action would typically be

Re: Move --data-checksums to common options in initdb --help

2021-01-02 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Fri, Jan 01, 2021 at 08:34:34PM +0100, Michael Banck wrote: > > I think enough people use data checksums these days that it warrants to > > be moved into the "normal part", like in the attached. > > +1. Let's see first what others

Re: crash recovery vs partially written WAL

2020-12-31 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Dec 30, 2020 at 12:52:46PM -0800, Andres Freund wrote: > > A question from a colleague made me wonder if there are scenarios where > > two subsequent crashes could lead to wrong WAL to be applied. > > > > Imagine the following

Re: Proposed patch for key management

2020-12-31 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Dec 31, 2020 at 11:11:11AM +0100, Fabien COELHO wrote: > > > I am not sure what else I can add to this discussion. Having something > > > that is completely external might be a nice option, but I don't think it > > > is the common

Re: Proposed patch for key managment

2020-12-31 Thread Stephen Frost
Greetings, * Fabien COELHO (coe...@cri.ensmp.fr) wrote: > >>The implementations should not have to be in any particular language: Shell, > >>Perl, Python, C should be possible. > > > >I disagree that it makes any sense to pass actual encryption out to a > >shell script. > > Yes, sure. I'm

Re: Proposed patch for key management

2020-12-31 Thread Stephen Frost
Greetings, * Fabien COELHO (coe...@cri.ensmp.fr) wrote: > >>The API should NOT make assumptions about the cryptographic design, what > >>depends about what, where things are stored… ISTM that Pg should only care > >>about naming keys, holding them when created/retrieved (but not create > >>them),

Re: Proposed patch for key managment

2020-12-31 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Dec 30, 2020 at 06:49:34PM -0500, Stephen Frost wrote: > > The API to fetch the KEK doesn't care at all about where it's stored or > > how it's derived or anything like that. There's a relatively small > > chang

Re: Proposed patch for key managment

2020-12-30 Thread Stephen Frost
Greetings, * Fabien COELHO (coe...@cri.ensmp.fr) wrote: > I think that an API should be carefully thought about, without assumption > about the underlying cryptography (algorithm, key lengths, modes, how keys > are derived and stored, and so on), and its usefulness be demonstrated by > actually

Re: [PATCH] Simplify permission checking logic in user.c

2020-12-30 Thread Stephen Frost
Greetings, * Paul Martinez (paul...@google.com) wrote: > You've identified exactly the problem we're running into -- we want to > allow customers, who aren't superusers, to create replication roles. This is also where it's probably useful to think about what the impact of that is- after all,

Re: [PATCH] Simplify permission checking logic in user.c

2020-12-30 Thread Stephen Frost
Greetings, * Michael Paquier (mich...@paquier.xyz) wrote: > On Tue, Dec 29, 2020 at 02:26:19PM -0600, Paul Martinez wrote: > > The checks for whether the current user can create a user with the > > SUPERUSER, > > REPLICATION, or BYPASSRLS attributes are chained together using if/else-if, > >

<    2   3   4   5   6   7   8   9   10   11   >