Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-07-07 Thread Thomas Munro
On Sat, Jul 6, 2019 at 2:42 AM Antonin Houska wrote: > I've reworked the way key is passed to postmaster (v04-0003-...) so it's > easier to call the encryption key command interactively, adjusted pg_upgrade > so that it preserves database, tablespace and relfilenode OIDs (v04-0014-...), >

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-07-05 Thread Bruce Momjian
On Tue, Jun 25, 2019 at 02:28:00PM +0200, Peter Eisentraut wrote: > On 2019-06-17 11:23, Antonin Houska wrote: > > I'm thinking how to teach postmaster to accept FEBE protocol connections > > temporarily, just to receive the key. The user applications like pg_ctl, > > initdb or pg_upgrade would

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-26 Thread Peter Eisentraut
On 2019-06-25 15:39, Robert Haas wrote: > On Tue, Jun 25, 2019 at 8:28 AM Peter Eisentraut > wrote: >> How are the requirements here different from ssl_passphrase_command? >> Why do we need a new mechanism? > > I don't think that the requirements are different, and I don't think > we need a new

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-25 Thread Robert Haas
On Tue, Jun 25, 2019 at 8:28 AM Peter Eisentraut wrote: > How are the requirements here different from ssl_passphrase_command? > Why do we need a new mechanism? I don't think that the requirements are different, and I don't think we need a new mechanism. I am curious exactly how you would set

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-25 Thread Peter Eisentraut
On 2019-06-17 11:23, Antonin Houska wrote: > I'm thinking how to teach postmaster to accept FEBE protocol connections > temporarily, just to receive the key. The user applications like pg_ctl, > initdb or pg_upgrade would retrieve the key / password from the DBA, then > start postmaster and send

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-17 Thread Antonin Houska
Bruce Momjian wrote: > On Mon, Jun 3, 2019 at 12:04:54PM -0400, Robert Haas wrote: > > > > What I'm talking about here is that it also has to be reasonably > > possible to write an encryption key command that does something > > useful. I don't have a really clear vision for how that's going

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-14 Thread Bruce Momjian
On Thu, Jun 13, 2019 at 09:14:13PM -0400, Bruce Momjian wrote: > On Mon, Jun 3, 2019 at 12:04:54PM -0400, Robert Haas wrote: > > > > What I'm talking about here is that it also has to be reasonably > > possible to write an encryption key command that does something > > useful. I don't have a

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-13 Thread Bruce Momjian
On Mon, Jun 3, 2019 at 12:04:54PM -0400, Robert Haas wrote: > > What I'm talking about here is that it also has to be reasonably > possible to write an encryption key command that does something > useful. I don't have a really clear vision for how that's going to > work. Nobody wants the

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-06 Thread Pavlo Golub
Greetings, Antonin. You wrote 2019-06-05, 15:32: > Robert Haas wrote: >> On Fri, May 31, 2019 at 2:59 AM Antonin Houska wrote: >> > > Sounds good. I'm not quite sure how this is going to work. Ideally >> > > you'd like the encryption key command to fetch the key from something >> > > like

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-05 Thread Antonin Houska
Robert Haas wrote: > On Fri, May 31, 2019 at 2:59 AM Antonin Houska wrote: > > > Sounds good. I'm not quite sure how this is going to work. Ideally > > > you'd like the encryption key command to fetch the key from something > > > like ssh-agent, or maybe pop up a window on the user's terminal

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-06-03 Thread Robert Haas
On Fri, May 31, 2019 at 2:59 AM Antonin Houska wrote: > > Sounds good. I'm not quite sure how this is going to work. Ideally > > you'd like the encryption key command to fetch the key from something > > like ssh-agent, or maybe pop up a window on the user's terminal with a > > key prompt. Just

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-05-31 Thread Antonin Houska
Robert Haas wrote: > On Tue, May 28, 2019 at 11:27 AM Antonin Houska wrote: > > We thought that it's more convenient for administrator to enter password. To > > achieve that, we can instead tell the admin how to use the key derivation > > tool > > (pg_keysetup) and send its output to postgres.

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-05-29 Thread Robert Haas
On Tue, May 28, 2019 at 11:27 AM Antonin Houska wrote: > We thought that it's more convenient for administrator to enter password. To > achieve that, we can instead tell the admin how to use the key derivation tool > (pg_keysetup) and send its output to postgres. So yes, it's possible to always >

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-05-28 Thread Antonin Houska
Thanks of another round of review. Robert Haas wrote: > On Wed, Apr 24, 2019 at 10:48 AM Antonin Houska wrote: > > Attached is the next version. It tries to address various problems pointed > > out > > upthread, including documentation. > > It's not immediately evident to me, perhaps because

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-05-22 Thread Robert Haas
On Wed, Apr 24, 2019 at 10:48 AM Antonin Houska wrote: > Attached is the next version. It tries to address various problems pointed out > upthread, including documentation. It's not immediately evident to me, perhaps because I haven't looked at this stuff in a while, what the purpose of 0001 and

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-12 Thread Bruce Momjian
On Sat, Apr 6, 2019 at 03:29:13PM +0200, Antonin Houska wrote: > Robert Haas wrote: > > > On Fri, Apr 5, 2019 at 11:22 AM Antonin Houska wrote: > > > > Wouldn't Tom's proposal to use a stream cipher fix all this? > > > > > > Yes it would make the extra alignment unnecessary, but our solution

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-06 Thread Antonin Houska
Robert Haas wrote: > On Fri, Apr 5, 2019 at 11:22 AM Antonin Houska wrote: > > > Wouldn't Tom's proposal to use a stream cipher fix all this? > > > > Yes it would make the extra alignment unnecessary, but our solution tries to > > meet specific requirements of disk encryption. Stream cipher

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-05 Thread Robert Haas
On Fri, Apr 5, 2019 at 11:22 AM Antonin Houska wrote: > > Wouldn't Tom's proposal to use a stream cipher fix all this? > > Yes it would make the extra alignment unnecessary, but our solution tries to > meet specific requirements of disk encryption. Stream cipher appears to be > incompatible with

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-05 Thread Antonin Houska
Robert Haas wrote: > On Thu, Mar 21, 2019 at 7:46 AM Antonin Houska wrote: > > Nevertheless, with the current version of our patch, PG should be resistant > > against such a partial write anyway because we chose to align XLOG records > > to > > 16 bytes (as long as the encryption is enabled)

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Robert Haas
On Thu, Apr 4, 2019 at 11:52 AM Antonin Houska wrote: > Robert Haas wrote: > > I'm willing to put some effort into trying to get this into v13 if > > you're willing to keep hacking on it, but there's probably a fair > > amount to do and a year can go by in a hurry. > > That'd be appreciated,

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Antonin Houska
Robert Haas wrote: > I'm willing to put some effort into trying to get this into v13 if > you're willing to keep hacking on it, but there's probably a fair > amount to do and a year can go by in a hurry. That'd be appreciated, especially by my boss. It doesn't seem likely that in this situation

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Robert Haas
On Thu, Apr 4, 2019 at 9:57 AM Antonin Houska wrote: > I think I finally understand. Originally I thought the question is how to > compute correct page checksum while the hint bits can be changed w/o exclusive > lock on the buffer. Now I realize that it's more about *recovery*: if the hint > bit

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-04-04 Thread Antonin Houska
Robert Haas wrote: > On Thu, Mar 14, 2019 at 9:26 AM Antonin Houska wrote: > > > Hint bits also rely on this principle. I thought there might be some > > > interaction between this work and wal_log_hints for this reason, but I see > > > nothing of that sort in the patch. > > > > I'm not sure

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-21 Thread Robert Haas
On Thu, Mar 21, 2019 at 7:46 AM Antonin Houska wrote: > Nevertheless, with the current version of our patch, PG should be resistant > against such a partial write anyway because we chose to align XLOG records to > 16 bytes (as long as the encryption is enabled) for the following reasons: > > If

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-21 Thread Antonin Houska
Tom Lane wrote: > Robert Haas writes: > > If the WAL *is* encrypted, the state at this point is that the block > > is unreadable, because the first 4kB of the block is the first half of > > the bits that resulted from encrypting 8kB of data that includes the > > new record, and the second 4kB

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-15 Thread Robert Haas
On Fri, Mar 15, 2019 at 5:10 PM Tom Lane wrote: > ISTM that this is only a problem if you choose the wrong encryption > method. One not-wrong encryption method is to use a stream cipher > --- maybe that's not the exact right technical term, but anyway > I'm talking about a method which

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-15 Thread Tom Lane
Robert Haas writes: > If the WAL *is* encrypted, the state at this point is that the block > is unreadable, because the first 4kB of the block is the first half of > the bits that resulted from encrypting 8kB of data that includes the > new record, and the second 4kB of the block is the second

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-15 Thread Robert Haas
On Thu, Mar 14, 2019 at 9:26 AM Antonin Houska wrote: > The point of #ifdef USE_ENCRYPTION is that we rely on OpenSSL, so the code > needs to compile even w/o OpenSSL (of course the encryption won't be enabled > in that case). I'll try to reduce the use of this construct only to the code > blocks

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-14 Thread Antonin Houska
Thanks for feedback! Robert Haas wrote: > However, this is also quite invasive. It changes a lot of code and it > doesn't do so in a very predictable way. It's not like you went > through and replaced every call to write() with a call to > SpecialEncyptionMagicWrite(). Rather, there are new

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2019-03-01 Thread Robert Haas
On Fri, Sep 14, 2018 at 10:04 AM Antonin Houska wrote: > Toshi Harada wrote: > > Even if you apply "data-at-rest-encryption-wip-2018.07.25.patch" to the > > master branch, > > the same patch error will occur. > > The version attached to this email should be applicable to the current > branch.

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2018-08-14 Thread Toshi Harada
Hi. Even if you apply "data-at-rest-encryption-wip-2018.07.25.patch" to the master branch, the same patch error will occur. (Patch error of pg_rewind was eliminated) patching file src/backend/replication/logical/reorderbuffer.c Hunk #9 FAILED at 2464. patching file

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2018-08-13 Thread Antonin Houska
Toshi Harada wrote: > Hi. > > If "data-at-rest-encryption-wip-2018.07.25.patch" is applied to PostgreSQL > 11-beta3 released last week, > patch execution will fail as follows. > > patching file src/backend/replication/logical/reorderbuffer.c > Hunk #9 FAILED at 2464. > > 1 out of 7

"WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

2018-08-13 Thread Toshi Harada
Hi. If "data-at-rest-encryption-wip-2018.07.25.patch" is applied to PostgreSQL 11-beta3 released last week, patch execution will fail as follows. patching file src/backend/replication/logical/reorderbuffer.c Hunk #9 FAILED at 2464. 1 out of 7 hunks FAILED -- saving rejects to file