Re: Key management with tests

2021-04-06 Thread Neil Chen
Hi Bruce, I went through these patches and executed the test script you added for the KMS section, which looks all good. This is a point that looks like a bug - in patch 10, you changed the location and use of *RelFileNodeSkippingWAL()*, but the modified code logic seems different from the

Re: Key management with tests

2021-03-22 Thread Bruce Momjian
On Mon, Mar 22, 2021 at 08:38:37PM -0400, Bruce Momjian wrote: > > This particular patch (introducing the RelationIsPermanent() macro) > > seems like it'd be a nice thing to commit independently of the rest, > > reducing the size of this patch set..? > > Committed as suggested. Also, I have

Re: Key management with tests

2021-03-22 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 11:31:34AM -0400, Stephen Frost wrote: > > src/backend/access/gist/gistutil.c | 2 +- > > src/backend/access/heap/heapam_handler.c | 2 +- > > src/backend/catalog/pg_publication.c | 2 +- > > src/backend/commands/tablecmds.c | 10 +- > >

Re: Key management with tests

2021-03-18 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 01:46:28PM -0400, Stephen Frost wrote: > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > This caught my attention because a comment says "encryption does not > > support WAL-skipped relations", but there's no direct change to the > > definition of

Re: Key management with tests

2021-03-18 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 02:37:43PM -0300, Álvaro Herrera wrote: > On 2021-Mar-18, Stephen Frost wrote: > > This is discussed in src/backend/access/transam/README, specifically the > > section that talks about Skipping WAL for New RelFileNode. Basically, > > it's the 'wal_level=minimal'

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > On 2021-Mar-18, Stephen Frost wrote: > > > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > > > it "relations that are not WAL-logged"? I suppose we

Re: Key management with tests

2021-03-18 Thread Alvaro Herrera
On 2021-Mar-18, Stephen Frost wrote: > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > > it "relations that are not WAL-logged"? I suppose we already have a > > term for this; I'm not sure it's a good idea to invent a

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > Patch 10 uses the term "WAL-skip relations". What does that mean? Is > it "relations that are not WAL-logged"? I suppose we already have a > term for this; I'm not sure it's a good idea to invent a different term > that is only

Re: Key management with tests

2021-03-18 Thread Alvaro Herrera
Patch 10 uses the term "WAL-skip relations". What does that mean? Is it "relations that are not WAL-logged"? I suppose we already have a term for this; I'm not sure it's a good idea to invent a different term that is only used in this new place. -- Álvaro Herrera

Re: Key management with tests

2021-03-18 Thread Bruce Momjian
On Thu, Mar 18, 2021 at 11:31:34AM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote: > > > I have made significant progress on the cluster file encryption feature so > > > it is time for me to

Re: Key management with tests

2021-03-18 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Mar 11, 2021 at 10:31:28PM -0500, Bruce Momjian wrote: > > I have made significant progress on the cluster file encryption feature so > > it is time for me to post a new set of patches. > > Here is a rebase, to keep the cfbot green.

Re: Key management with tests

2021-02-07 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 07:53:18PM -0500, Bruce Momjian wrote: > On Fri, Feb 5, 2021 at 05:21:22PM -0500, Stephen Frost wrote: > > > I disagree. If we only warn about some parts, attackers will just > > > attack other parts. It will also give users a false sense of security. > > > If you can

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 05:21:22PM -0500, Stephen Frost wrote: > > I disagree. If we only warn about some parts, attackers will just > > attack other parts. It will also give users a false sense of security. > > If you can get the keys, it doesn't matter if there is one or ten ways > > of

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote: > > > I looked further. First, I don't think we are going to be able to > > > protect at all against users who have _write_ access on the OS running > > > Postgres. It would be

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote: > > I looked further. First, I don't think we are going to be able to > > protect at all against users who have _write_ access on the OS running > > Postgres. It would be too easy to just read process memory, or modify > >

Re: Key management with tests

2021-02-05 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote: > > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > > I doubt anyone would actually stipulate that they *guarantee* detection > > > of malicious writes, and I

Re: Key management with tests

2021-02-05 Thread Bruce Momjian
On Wed, Feb 3, 2021 at 01:16:32PM -0500, Bruce Momjian wrote: > On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > I doubt anyone would actually stipulate that they *guarantee* detection > > of malicious writes, and I don't think we should either, but certainly > > the other

Re: Key management with tests

2021-02-03 Thread Bruce Momjian
On Wed, Feb 3, 2021 at 10:33:57AM -0500, Stephen Frost wrote: > > I am thinking group read-access might be a requirement for cluster file > > encryption to be effective. > > People certainly do use group read-access, but I don't see that as being > a requirement for cluster file encryption to be

Re: Key management with tests

2021-02-03 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Feb 1, 2021 at 07:47:57PM -0500, Bruce Momjian wrote: > > On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > The purpose of cluster file encryption is to prevent

Re: Key management with tests

2021-02-02 Thread Bruce Momjian
On Mon, Feb 1, 2021 at 07:47:57PM -0500, Bruce Momjian wrote: > On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > The purpose of cluster file encryption is to prevent users with read > > > access to the directories used to store

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Mon, Feb 1, 2021 at 06:31:32PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > The purpose of cluster file encryption is to prevent users with read > > access to the directories used to store database files and write-ahead > > log files from being able to

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Mon, Feb 1, 2021 at 06:34:53PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote: > > > I propose that we meet to discuss what approach we want to use to move TDE > > > forward.  We then start

Re: Key management with tests

2021-02-01 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote: > > I propose that we meet to discuss what approach we want to use to move TDE > > forward.  We then start a new thread with a proposal on the approach > > and finalize it via

Re: Key management with tests

2021-02-01 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 29, 2021 at 05:40:37PM -0500, Stephen Frost wrote: > > I hope it's pretty clear that I'm also very much in support of both this > > effort with the KMS and of TDE in general- TDE is specifically, > > Yes, thanks. I know we have

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Sat, Jan 30, 2021 at 08:23:11AM -0500, Tom Kincaid wrote: > I propose that we meet to discuss what approach we want to use to move TDE > forward.  We then start a new thread with a proposal on the approach > and finalize it via community consensus. I will invite Bruce, Stephen and > Masahiko to

Re: Key management with tests

2021-02-01 Thread Bruce Momjian
On Fri, Jan 29, 2021 at 05:05:06PM +0900, Masahiko Sawada wrote: > TBH I’m confused a bit about the recent situation of this patch, but > I Yes, it is easy to get confused. > can contribute to KMS work by discussing, writing, reviewing, and > testing the patch. Also, I can work on the data

Re: Key management with tests

2021-01-31 Thread Moon, Insung
Dear All. Thank you for all opinions and discussions regarding the KMS/TDE function. First of all, to get to the point of this email, I want to participate in anything I can do (review or development) when TDE related development is in progress. If there is a meeting related to it, I can't

Re: Key management with tests

2021-01-30 Thread Tom Kincaid
Thanks Stephen, Bruce and Masahiko, > > discussions so far and the point behind the design so that everyone > > can understand why this feature is designed in that way. To do that, > > it might be a good start to sort the wiki page since it has data > > encryption part, KMS, and ToDo mixed. > >

Re: Key management with tests

2021-01-29 Thread Stephen Frost
Greetings, * Masahiko Sawada (sawada.m...@gmail.com) wrote: > On Fri, Jan 29, 2021 at 5:22 AM Bruce Momjian wrote: > > On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote: > > > I would also like to add a "not wanted" entry for this feature on the > > > TODO list, baaed on the

Re: Key management with tests

2021-01-29 Thread Masahiko Sawada
On Fri, Jan 29, 2021 at 5:22 AM Bruce Momjian wrote: > > On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote: > > I would also like to add a "not wanted" entry for this feature on the > > TODO list, baaed on the feature's limited usefulness, but I already > > asked about that

Re: Key management with tests

2021-01-28 Thread Bruce Momjian
On Thu, Jan 28, 2021 at 02:41:09PM -0500, Tom Kincaid wrote: > I would also like to add a "not wanted" entry for this feature on the > TODO list, baaed on the feature's limited usefulness, but I already > asked about that and no one seems to feel we don't want it. > > > I want to

Re: Key management with tests

2021-01-28 Thread Tom Kincaid
Hello, > > I don't think it makes sense to think about committing this to v14. I > > believe it only makes sense if we have a TDE patch that is relatively > > close to committable that can be used with it. I also don't think that > > this patch is in good enough shape to commit yet in terms of

Re: Key management with tests

2021-01-27 Thread Bruce Momjian
On Tue, Jan 26, 2021 at 05:53:01PM -0500, Bruce Momjian wrote: > On Tue, Jan 26, 2021 at 03:24:30PM -0500, Robert Haas wrote: > > I'm wondering whether you've considered storing all the keys in one > > file instead of a file per key. The reason I ask is that it seems to > > me that the key

Re: Key management with tests

2021-01-26 Thread Robert Haas
On Tue, Jan 26, 2021 at 11:15 AM Bruce Momjian wrote: > This version fixes OpenSSL detection and improves docs for initdb > interactions. Hi, I'm wondering whether you've considered storing all the keys in one file instead of a file per key. The reason I ask is that it seems to me that the key

Re: Key management with tests

2021-01-25 Thread Bruce Momjian
On Mon, Jan 25, 2021 at 08:12:01PM -0300, Álvaro Herrera wrote: > In patch 1, > > * The docs are not clear on what happens if --auth-prompt is not given > but an auth prompt is required for the program to work. Should it exit > with a status other than 0? Uh, I think the docs talk about this:

Re: Key management with tests

2021-01-25 Thread Alvaro Herrera
In patch 1, * The docs are not clear on what happens if --auth-prompt is not given but an auth prompt is required for the program to work. Should it exit with a status other than 0? * BootStrapKmgr claims it is called by initdb, but that doesn't seem to be the case. * Also, BootStrapKmgr is

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 04:38:47PM -0500, Robert Haas wrote: > To me, it wouldn't make sense to commit a full README for a TDE > feature that we don't have yet with a key management patch, but the > way that they'll interact with each other has to be clear. The > doc/database-encryption.sgml file

Re: Key management with tests

2021-01-18 Thread Tom Kincaid
I met with Bruce and Stephen this afternoon to discuss the feedback we received so far (prior to Robert's note which I haven't fully digested yet) on this patch. Here is what we plan to do: 1) Bruce is going to gather all the details from the Wiki and build a README for the TDE Key Management

Re: Key management with tests

2021-01-18 Thread Robert Haas
On Mon, Jan 18, 2021 at 2:00 PM Tom Kincaid wrote: > Some of the missing things you mention above are about the design of > TDE feature in general. However, this patch is about Key Management > which is going part of the larger TDE feature. So it feels as though > there is the need for a general

Re: Key management with tests

2021-01-18 Thread Andres Freund
On 2021-01-18 13:58:20 -0500, Bruce Momjian wrote: > On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote: > > Personally, but I admit that there's legitimate reasons to differ on > > that note, I don't think it's reasonable for a feature this invasive to > > commit preliminary patches

Re: Key management with tests

2021-01-18 Thread Tom Kincaid
> > I have to admit I was kind of baffled that the wiki page wasn't > > sufficient, because it is one of the longest Postgres feature > > explanations I have seen, but I now think the missing part is tying > > the wiki contents to the code implementation. If that is it, please > > confirm. If it

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 09:42:54AM -0800, Andres Freund wrote: > Personally, but I admit that there's legitimate reasons to differ on > that note, I don't think it's reasonable for a feature this invasive to > commit preliminary patches without the major subsequent patches being in > a shape that

Re: Key management with tests

2021-01-18 Thread Andres Freund
Hi, On 2021-01-18 12:06:35 -0500, Bruce Momjian wrote: > On Mon, Jan 18, 2021 at 10:50:37AM -0500, Bruce Momjian wrote: > > OK, I looked at that and it is good, and I see my patch is missing that. > > Are people looking for me to take the wiki content, expand on it and tie > > it to the code that

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Mon, Jan 18, 2021 at 10:50:37AM -0500, Bruce Momjian wrote: > OK, I looked at that and it is good, and I see my patch is missing that. > Are people looking for me to take the wiki content, expand on it and tie > it to the code that will be applied, or something else like all the > various

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Sat, Jan 16, 2021 at 10:58:47PM -0800, Andres Freund wrote: > Hi, > > On 2021-01-17 11:54:57 +0530, Amit Kapila wrote: > > On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid > > wrote: > > > Admittedly I am a novice on this topic, and the majority of the > > > PostgreSQL source code, however I am

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Sun, Jan 17, 2021 at 11:54:57AM +0530, Amit Kapila wrote: > > Is there a design document for a Postgres feature of this size and > > scope that people feel would serve as a good example? Alternatively, > > is there a design document template that has been successfully used in > > the past? > >

Re: Key management with tests

2021-01-18 Thread Bruce Momjian
On Sun, Jan 17, 2021 at 07:50:13PM -0500, Robert Haas wrote: > On Fri, Jan 15, 2021 at 7:56 PM Andres Freund wrote: > > I think that's not at all acceptable. I don't mind hashing out details > > on calls / off-list, but the design needs to be public, documented, and > > reviewable. And if it's

Re: Key management with tests

2021-01-17 Thread Robert Haas
On Fri, Jan 15, 2021 at 7:56 PM Andres Freund wrote: > I think that's not at all acceptable. I don't mind hashing out details > on calls / off-list, but the design needs to be public, documented, and > reviewable. And if it's something the community can't understand, then > it can't get in.

Re: Key management with tests

2021-01-16 Thread Andres Freund
Hi, On 2021-01-17 11:54:57 +0530, Amit Kapila wrote: > On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid wrote: > > Admittedly I am a novice on this topic, and the majority of the > > PostgreSQL source code, however I am hopeful enough (those of you who > > know me understand that I suffer from

Re: Key management with tests

2021-01-16 Thread Amit Kapila
On Sun, Jan 17, 2021 at 5:38 AM Tom Kincaid wrote: > > > > > I think that's not at all acceptable. I don't mind hashing out details > > > > on calls / off-list, but the design needs to be public, documented, and > > > > reviewable. And if it's something the community can't understand, then > > >

Re: Key management with tests

2021-01-16 Thread Tom Kincaid
> > > I think that's not at all acceptable. I don't mind hashing out details > > > on calls / off-list, but the design needs to be public, documented, and > > > reviewable. And if it's something the community can't understand, then > > > it can't get in. We're going to have to maintain this going

Re: Key management with tests

2021-01-15 Thread Michael Paquier
On Fri, Jan 15, 2021 at 08:20:36PM -0800, Andres Freund wrote: > On 2021-01-15 20:49:10 -0500, Bruce Momjian wrote: >> What Perl tap tests run initdb and manage the cluster? I didn't find >> any. > > find . -name '*.pl'|xargs grep 'use PostgresNode;' > > should give you a nearly complete list.

Re: Key management with tests

2021-01-15 Thread Andres Freund
Hi, On 2021-01-15 20:49:10 -0500, Bruce Momjian wrote: > On Fri, Jan 15, 2021 at 04:56:24PM -0800, Andres Freund wrote: > > On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote: > > > You have to understand cryptography and Postgres internals to understand > > > the design, and I don't think it is

Re: Key management with tests

2021-01-15 Thread Bruce Momjian
On Fri, Jan 15, 2021 at 04:56:24PM -0800, Andres Freund wrote: > On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote: > > You have to understand cryptography and Postgres internals to understand > > the design, and I don't think it is realistic to explain that all to the > > community. We did much

Re: Key management with tests

2021-01-15 Thread Andres Freund
Hi, On 2021-01-15 19:21:32 -0500, Bruce Momjian wrote: > On Fri, Jan 15, 2021 at 02:37:56PM -0800, Andres Freund wrote: > > On 2021-01-15 16:47:19 -0500, Bruce Momjian wrote: > > > > I am not even sure there is a consensus on the design, without which > > > > any commit is always premature. > > >

Re: Key management with tests

2021-01-15 Thread Andres Freund
Hi, On 2021-01-15 16:47:19 -0500, Bruce Momjian wrote: > On Fri, Jan 15, 2021 at 04:23:22PM -0500, Robert Haas wrote: > > On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian wrote: > > I don't think that's appropriate. Several prominent community members > > have told you that the patch, as committed

Re: Key management with tests

2021-01-15 Thread Bruce Momjian
On Fri, Jan 15, 2021 at 04:59:17PM -0500, Robert Haas wrote: > On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian wrote: > > If people want changes, I need to hear about it here. I have address > > everything people have mentioned in these threads so far. > > That does not match my perception of the

Re: Key management with tests

2021-01-15 Thread David G. Johnston
On Fri, Jan 15, 2021 at 2:59 PM Robert Haas wrote: > On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian wrote: > > If people want changes, I need to hear about it here. I have address > > everything people have mentioned in these threads so far. > > That does not match my perception of the

Re: Key management with tests

2021-01-15 Thread Robert Haas
On Fri, Jan 15, 2021 at 4:47 PM Bruce Momjian wrote: > If people want changes, I need to hear about it here. I have address > everything people have mentioned in these threads so far. That does not match my perception of the situation. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Key management with tests

2021-01-15 Thread Bruce Momjian
On Fri, Jan 15, 2021 at 04:23:22PM -0500, Robert Haas wrote: > On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian wrote: > > I am planning to apply this next week. > > I don't think that's appropriate. Several prominent community members > have told you that the patch, as committed the first time,

Re: Key management with tests

2021-01-15 Thread Robert Haas
On Fri, Jan 15, 2021 at 3:49 PM Bruce Momjian wrote: > I am planning to apply this next week. I don't think that's appropriate. Several prominent community members have told you that the patch, as committed the first time, needed a lot more work. There hasn't been enough time between then and

Re: Key management with tests

2021-01-13 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:46:53PM -0500, Bruce Momjian wrote: > On Tue, Jan 12, 2021 at 01:15:44PM -0500, Bruce Momjian wrote: > > Well, we have eight unused bits in the IV, so we could just increment > > that for every hint bit change that uses the same LSN, and then force a > > dummy WAL record

Re: Key management with tests

2021-01-12 Thread Neil Chen
Thank you for your reply, On Wed, Jan 13, 2021 at 12:08 AM Stephen Frost wrote: > > No, we can't 'modify the page format as we wish'- if we change away from > using a C structure then we're going to be modifying quite a bit of > code which otherwise doesn't need to be changed. The proposed

Re: Key management with tests

2021-01-12 Thread Andres Freund
Hi, On 2021-01-11 20:12:00 +0900, Masahiko Sawada wrote: > diff --git a/contrib/bloom/blinsert.c b/contrib/bloom/blinsert.c > index 32b5d62e1f..d474af753c 100644 > --- a/contrib/bloom/blinsert.c > +++ b/contrib/bloom/blinsert.c > @@ -177,6 +177,7 @@ blbuildempty(Relation index) >*

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:57:11PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > Well, we have eight unused bits in the IV, so we

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > Well, we have eight unused bits in the IV, so we could just increment > > > that for every hint bit change that uses the same

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:44:05PM -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > Well, we have eight unused bits in the IV, so we could just increment > > that for every hint bit change that uses the same LSN, and then force a > > dummy WAL record when that 8-bit

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:15:44PM -0500, Bruce Momjian wrote: > On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > I don't think there's any doubt that we need to make sure that the IV is > > distinct and advancing the LSN to get a new one when needed for this > > case seems like

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > > I think one big question is that, since we are using a streaming cipher, > > > do we care about hint bit changes showing to users? I actually don't > > > know. If we do,

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 01:11:29PM -0500, Stephen Frost wrote: > > I think one big question is that, since we are using a streaming cipher, > > do we care about hint bit changes showing to users? I actually don't > > know. If we do, some kind of dummy LSN record might be required, as you > >

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote: > > > This says: > > > > > > > > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements > > > > > > wal_log_hints will be enabled

Re: Key management with tests

2021-01-12 Thread Andres Freund
On 2021-01-12 13:03:14 -0500, Bruce Momjian wrote: > I think one big question is that, since we are using a streaming cipher, > do we care about hint bit changes showing to users? I actually don't > know. If we do, some kind of dummy LSN record might be required, as you > suggested. That'd lead

Re: Key management with tests

2021-01-12 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 09:40:53PM +0900, Masahiko Sawada wrote: > > This says: > > > > > > https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Other_requirements > > > > wal_log_hints will be enabled automatically in encryption mode. > > > > Does that help? > > IIUC it

Re: Key management with tests

2021-01-12 Thread Stephen Frost
Greetings, * Neil Chen (carpenter.nail...@gmail.com) wrote: > On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost wrote: > > This is an interesting question but ultimately I don't think we should > > be looking at this from the perspective of allowing arbitrary changes to > > the page format. The

Re: Key management with tests

2021-01-12 Thread Masahiko Sawada
On Tue, Jan 12, 2021 at 11:09 AM Bruce Momjian wrote: > > On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote: > > On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > > Right, or ensure that the actual IV used is distinct (such as by using > > > another bit in the IV to

Re: Key management with tests

2021-01-11 Thread Neil Chen
Hi Stephen, On Tue, Jan 12, 2021 at 10:47 AM Stephen Frost wrote: > > This is an interesting question but ultimately I don't think we should > be looking at this from the perspective of allowing arbitrary changes to > the page format. The challenge is that much of the page format, today, > is

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote: > On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > Right, or ensure that the actual IV used is distinct (such as by using > > another bit in the IV to distinguish logged-vs-unlogged), but it seems > > saner to just use a

Re: Key management with tests

2021-01-11 Thread Masahiko Sawada
On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > > > Although, another approach and one that I've discussed a bit with Bruce, > > > is to have more keys- such as a

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 02:19:22PM -0500, Stephen Frost wrote: > > outputs from the GCM encryption and is what provides the integrity / > > authentication of the encrypted data to be able to detect if it's been > > modified. Unfortunately,

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 02:19:22PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote: > > > Yes, and it avoids the issue of using a single key for too much, which > > > is also a concern. The

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote: > > Yes, and it avoids the issue of using a single key for too much, which > > is also a concern. The remaining larger issues are to figure out a > > place to put the tag for

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 01:23:27PM -0500, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > > > Although, another approach and one that I've discussed a bit with Bruce, > > > is to have more keys-

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > > Although, another approach and one that I've discussed a bit with Bruce, > > is to have more keys- such as a key for temporary files, and perhaps > > even a key for logged

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 12:54:49PM -0500, Stephen Frost wrote: > Although, another approach and one that I've discussed a bit with Bruce, > is to have more keys- such as a key for temporary files, and perhaps > even a key for logged relations and a different for unlogged.. Or Yes, we have to

Re: Key management with tests

2021-01-11 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Jan 11, 2021 at 08:12:00PM +0900, Masahiko Sawada wrote: > > Looking at the patch, it supports three algorithms but only > > PG_CIPHER_AES_KWP is used in the core for now: > > > > +/* > > + * Supported symmetric encryption algorithm.

Re: Key management with tests

2021-01-11 Thread Bruce Momjian
On Mon, Jan 11, 2021 at 08:12:00PM +0900, Masahiko Sawada wrote: > On Sun, Jan 10, 2021 at 11:51 PM Bruce Momjian wrote: > > OK, here they are with numeric prefixes. It was actually tricky to > > figure out how to create a squashed format-patch based on another branch. > > Thank you for

Re: Key management with tests

2021-01-11 Thread Masahiko Sawada
On Sun, Jan 10, 2021 at 11:51 PM Bruce Momjian wrote: > > On Sun, Jan 10, 2021 at 06:04:12PM +1300, Thomas Munro wrote: > > On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian wrote: > > > Does anyone know why the cfbot applied the patch listed second first > > > here? > > > > > >

Re: Key management with tests

2021-01-09 Thread Thomas Munro
On Sun, Jan 10, 2021 at 3:45 PM Bruce Momjian wrote: > Does anyone know why the cfbot applied the patch listed second first > here? > > http://cfbot.cputube.org/patch_31_2925.log > > Specifically, it applied hex..key.diff.gz before hex.diff.gz. I assumed > it would apply attachments in

Re: Key management with tests

2021-01-09 Thread Bruce Momjian
On Sat, Jan 9, 2021 at 08:08:16PM -0500, Bruce Momjian wrote: > On Sat, Jan 9, 2021 at 01:17:36PM -0500, Bruce Momjian wrote: > > I know we are still working on the hex patch (dest-len) and the crypto > > tests, but I wanted to post this so people can see where we are, and we > > can get some

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 8, 2021 at 03:33:44PM -0500, Stephen Frost wrote: > > > Anyway, I think we need to figure out how to trim. The first part would > > > be to figure out whether we need 128 _and_ 256-bit tests, and then see > > > what items are

Re: Key management with tests

2021-01-08 Thread Bruce Momjian
On Fri, Jan 8, 2021 at 03:34:23PM -0500, Stephen Frost wrote: > > All the tests pass now. The current src/test directory is 19MB, and > > adding these tests takes it to 23MB, or a 20% increase. That seems like > > a lot. It is testing 128-bit and 256-bit keys --- should we do fewer > > tests,

Re: Key management with tests

2021-01-08 Thread Bruce Momjian
On Fri, Jan 8, 2021 at 03:33:44PM -0500, Stephen Frost wrote: > > No, I don't think so. Stephen imported the entire NIST test suite. It > > was so comperhensive, it detected several OpenSSL bugs for zero-length > > strings, which I already reported, but we would never be encrypting > >

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings Bruce, * Bruce Momjian (br...@momjian.us) wrote: > On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote: > > On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote: > > > I have completed the key management patch with tests created by Stephen > >

Re: Key management with tests

2021-01-08 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jan 7, 2021 at 04:08:49PM -0300, Álvaro Herrera wrote: > > On 2021-Jan-07, Bruce Momjian wrote: > > > > > All the tests pass now. The current src/test directory is 19MB, and > > > adding these tests takes it to 23MB, or a 20%

Re: Key management with tests

2021-01-07 Thread Bruce Momjian
On Thu, Jan 7, 2021 at 10:02:14AM -0500, Bruce Momjian wrote: > My next step is to add the high-level tests. Here is the high-level script, and the log output. I used the pg_upgrade test.sh as a model. It uses "CFE DEBUG" lines that are already in the code to compare the initdb encryption with

Re: Key management with tests

2021-01-07 Thread Bruce Momjian
On Thu, Jan 7, 2021 at 04:08:49PM -0300, Álvaro Herrera wrote: > On 2021-Jan-07, Bruce Momjian wrote: > > > All the tests pass now. The current src/test directory is 19MB, and > > adding these tests takes it to 23MB, or a 20% increase. That seems like > > a lot. It is testing 128-bit and

Re: Key management with tests

2021-01-07 Thread Alvaro Herrera
On 2021-Jan-07, Bruce Momjian wrote: > All the tests pass now. The current src/test directory is 19MB, and > adding these tests takes it to 23MB, or a 20% increase. That seems like > a lot. It is testing 128-bit and 256-bit keys --- should we do fewer > tests, or just test 256, or use gzip to

Re: Key management with tests

2021-01-07 Thread Bruce Momjian
On Fri, Jan 1, 2021 at 01:07:50AM -0500, Bruce Momjian wrote: > On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote: > > I have completed the key management patch with tests created by Stephen > > Frost. Original patch by Masahiko Sawada. It requires the hex > >

Re: Key management with tests

2020-12-31 Thread Bruce Momjian
On Thu, Dec 31, 2020 at 11:50:47PM -0500, Bruce Momjian wrote: > I have completed the key management patch with tests created by Stephen > Frost. Original patch by Masahiko Sawada. It requires the hex > reorganization patch first. The key patch is now 2.1MB because of the > tests,

Key management with tests

2020-12-31 Thread Bruce Momjian
I have completed the key management patch with tests created by Stephen Frost. Original patch by Masahiko Sawada. It requires the hex reorganization patch first. The key patch is now 2.1MB because of the tests, so attaching it here seems unwise: https://github.com/postgres/postgres

  1   2   >