Re: WIP: Data at rest encryption

On Wed, Sep 04, 2019 at 06:56:18AM +0200, Antonin Houska wrote: > This thread started later than our effort but important design questions are > being discussed there. So far there seems to be no consensus whether > full-instance encryption should be implemented first, so any effort spent on >

Re: WIP: Data at rest encryption

On Wed, 04 Sep 2019 00:56:15 +0800 Alvaro Herrera wrote On 2019-Aug-02, Shawn Wang wrote: > Hi Antonin, > It is very glad to see the new patch. I used the public patches a long time > ago. > I did some tests like the stream replication, much

Re: WIP: Data at rest encryption

Alvaro Herrera wrote: > On 2019-Aug-02, Shawn Wang wrote: > > > Hi Antonin, > > It is very glad to see the new patch. I used the public patches a long time > > ago. > > I did some tests like the stream replication, much data running, temporary > > files encryption. > > I found that there is

Re: WIP: Data at rest encryption

On Tue, Sep 3, 2019 at 10:21 PM Alvaro Herrera wrote: > On 2019-Aug-02, Shawn Wang wrote: > > > Hi Antonin, > > It is very glad to see the new patch. I used the public patches a long > time ago. > > I did some tests like the stream replication, much data running, > temporary files encryption. >

Re: WIP: Data at rest encryption

On 2019-Aug-02, Shawn Wang wrote: > Hi Antonin, > It is very glad to see the new patch. I used the public patches a long time > ago. > I did some tests like the stream replication, much data running, temporary > files encryption. > I found that there is an issue in the

Re: WIP: Data at rest encryption

The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, failed Spec compliant: tested, failed Documentation:not tested Hi Antonin, It is very glad to see the new patch. I used the

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Sat, Jul 6, 2019 at 2:42 AM Antonin Houska wrote: > I've reworked the way key is passed to postmaster (v04-0003-...) so it's > easier to call the encryption key command interactively, adjusted pg_upgrade > so that it preserves database, tablespace and relfilenode OIDs (v04-0014-...), >

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Tue, Jun 25, 2019 at 02:28:00PM +0200, Peter Eisentraut wrote: > On 2019-06-17 11:23, Antonin Houska wrote: > > I'm thinking how to teach postmaster to accept FEBE protocol connections > > temporarily, just to receive the key. The user applications like pg_ctl, > > initdb or pg_upgrade would

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On 2019-06-25 15:39, Robert Haas wrote: > On Tue, Jun 25, 2019 at 8:28 AM Peter Eisentraut > wrote: >> How are the requirements here different from ssl_passphrase_command? >> Why do we need a new mechanism? > > I don't think that the requirements are different, and I don't think > we need a new

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Tue, Jun 25, 2019 at 8:28 AM Peter Eisentraut wrote: > How are the requirements here different from ssl_passphrase_command? > Why do we need a new mechanism? I don't think that the requirements are different, and I don't think we need a new mechanism. I am curious exactly how you would set

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On 2019-06-17 11:23, Antonin Houska wrote: > I'm thinking how to teach postmaster to accept FEBE protocol connections > temporarily, just to receive the key. The user applications like pg_ctl, > initdb or pg_upgrade would retrieve the key / password from the DBA, then > start postmaster and send

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Bruce Momjian wrote: > On Mon, Jun 3, 2019 at 12:04:54PM -0400, Robert Haas wrote: > > > > What I'm talking about here is that it also has to be reasonably > > possible to write an encryption key command that does something > > useful. I don't have a really clear vision for how that's going

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Thu, Jun 13, 2019 at 09:14:13PM -0400, Bruce Momjian wrote: > On Mon, Jun 3, 2019 at 12:04:54PM -0400, Robert Haas wrote: > > > > What I'm talking about here is that it also has to be reasonably > > possible to write an encryption key command that does something > > useful. I don't have a

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Mon, Jun 3, 2019 at 12:04:54PM -0400, Robert Haas wrote: > > What I'm talking about here is that it also has to be reasonably > possible to write an encryption key command that does something > useful. I don't have a really clear vision for how that's going to > work. Nobody wants the

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Greetings, Antonin. You wrote 2019-06-05, 15:32: > Robert Haas wrote: >> On Fri, May 31, 2019 at 2:59 AM Antonin Houska wrote: >> > > Sounds good. I'm not quite sure how this is going to work. Ideally >> > > you'd like the encryption key command to fetch the key from something >> > > like

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Robert Haas wrote: > On Fri, May 31, 2019 at 2:59 AM Antonin Houska wrote: > > > Sounds good. I'm not quite sure how this is going to work. Ideally > > > you'd like the encryption key command to fetch the key from something > > > like ssh-agent, or maybe pop up a window on the user's terminal

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Fri, May 31, 2019 at 2:59 AM Antonin Houska wrote: > > Sounds good. I'm not quite sure how this is going to work. Ideally > > you'd like the encryption key command to fetch the key from something > > like ssh-agent, or maybe pop up a window on the user's terminal with a > > key prompt. Just

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Robert Haas wrote: > On Tue, May 28, 2019 at 11:27 AM Antonin Houska wrote: > > We thought that it's more convenient for administrator to enter password. To > > achieve that, we can instead tell the admin how to use the key derivation > > tool > > (pg_keysetup) and send its output to postgres.

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Tue, May 28, 2019 at 11:27 AM Antonin Houska wrote: > We thought that it's more convenient for administrator to enter password. To > achieve that, we can instead tell the admin how to use the key derivation tool > (pg_keysetup) and send its output to postgres. So yes, it's possible to always >

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Thanks of another round of review. Robert Haas wrote: > On Wed, Apr 24, 2019 at 10:48 AM Antonin Houska wrote: > > Attached is the next version. It tries to address various problems pointed > > out > > upthread, including documentation. > > It's not immediately evident to me, perhaps because

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Wed, Apr 24, 2019 at 10:48 AM Antonin Houska wrote: > Attached is the next version. It tries to address various problems pointed out > upthread, including documentation. It's not immediately evident to me, perhaps because I haven't looked at this stuff in a while, what the purpose of 0001 and

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Sat, Apr 6, 2019 at 03:29:13PM +0200, Antonin Houska wrote: > Robert Haas wrote: > > > On Fri, Apr 5, 2019 at 11:22 AM Antonin Houska wrote: > > > > Wouldn't Tom's proposal to use a stream cipher fix all this? > > > > > > Yes it would make the extra alignment unnecessary, but our solution

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Robert Haas wrote: > On Fri, Apr 5, 2019 at 11:22 AM Antonin Houska wrote: > > > Wouldn't Tom's proposal to use a stream cipher fix all this? > > > > Yes it would make the extra alignment unnecessary, but our solution tries to > > meet specific requirements of disk encryption. Stream cipher

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Fri, Apr 5, 2019 at 11:22 AM Antonin Houska wrote: > > Wouldn't Tom's proposal to use a stream cipher fix all this? > > Yes it would make the extra alignment unnecessary, but our solution tries to > meet specific requirements of disk encryption. Stream cipher appears to be > incompatible with

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Robert Haas wrote: > On Thu, Mar 21, 2019 at 7:46 AM Antonin Houska wrote: > > Nevertheless, with the current version of our patch, PG should be resistant > > against such a partial write anyway because we chose to align XLOG records > > to > > 16 bytes (as long as the encryption is enabled)

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Thu, Apr 4, 2019 at 11:52 AM Antonin Houska wrote: > Robert Haas wrote: > > I'm willing to put some effort into trying to get this into v13 if > > you're willing to keep hacking on it, but there's probably a fair > > amount to do and a year can go by in a hurry. > > That'd be appreciated,

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Robert Haas wrote: > I'm willing to put some effort into trying to get this into v13 if > you're willing to keep hacking on it, but there's probably a fair > amount to do and a year can go by in a hurry. That'd be appreciated, especially by my boss. It doesn't seem likely that in this situation

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Thu, Apr 4, 2019 at 9:57 AM Antonin Houska wrote: > I think I finally understand. Originally I thought the question is how to > compute correct page checksum while the hint bits can be changed w/o exclusive > lock on the buffer. Now I realize that it's more about *recovery*: if the hint > bit

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Robert Haas wrote: > On Thu, Mar 14, 2019 at 9:26 AM Antonin Houska wrote: > > > Hint bits also rely on this principle. I thought there might be some > > > interaction between this work and wal_log_hints for this reason, but I see > > > nothing of that sort in the patch. > > > > I'm not sure

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Thu, Mar 21, 2019 at 7:46 AM Antonin Houska wrote: > Nevertheless, with the current version of our patch, PG should be resistant > against such a partial write anyway because we chose to align XLOG records to > 16 bytes (as long as the encryption is enabled) for the following reasons: > > If

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Tom Lane wrote: > Robert Haas writes: > > If the WAL *is* encrypted, the state at this point is that the block > > is unreadable, because the first 4kB of the block is the first half of > > the bits that resulted from encrypting 8kB of data that includes the > > new record, and the second 4kB

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Fri, Mar 15, 2019 at 5:10 PM Tom Lane wrote: > ISTM that this is only a problem if you choose the wrong encryption > method. One not-wrong encryption method is to use a stream cipher > --- maybe that's not the exact right technical term, but anyway > I'm talking about a method which

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Robert Haas writes: > If the WAL *is* encrypted, the state at this point is that the block > is unreadable, because the first 4kB of the block is the first half of > the bits that resulted from encrypting 8kB of data that includes the > new record, and the second 4kB of the block is the second

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Thu, Mar 14, 2019 at 9:26 AM Antonin Houska wrote: > The point of #ifdef USE_ENCRYPTION is that we rely on OpenSSL, so the code > needs to compile even w/o OpenSSL (of course the encryption won't be enabled > in that case). I'll try to reduce the use of this construct only to the code > blocks

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Thanks for feedback! Robert Haas wrote: > However, this is also quite invasive. It changes a lot of code and it > doesn't do so in a very predictable way. It's not like you went > through and replaced every call to write() with a call to > SpecialEncyptionMagicWrite(). Rather, there are new

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

On Fri, Sep 14, 2018 at 10:04 AM Antonin Houska wrote: > Toshi Harada wrote: > > Even if you apply "data-at-rest-encryption-wip-2018.07.25.patch" to the > > master branch, > > the same patch error will occur. > > The version attached to this email should

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Hi. Even if you apply "data-at-rest-encryption-wip-2018.07.25.patch" to the master branch, the same patch error will occur. (Patch error of pg_rewind was eliminated) patching file src/backend/replication/logical/reorderbuffer.c Hunk #9 FAILED at 2464. patching file src/bin/

Re: "WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Toshi Harada wrote: > Hi. > > If "data-at-rest-encryption-wip-2018.07.25.patch" is applied to PostgreSQL > 11-beta3 released last week, > patch execution will fail as follows. > > patching file src/backend/replication/logical/reorderbuffer.c > Hunk

"WIP: Data at rest encryption" patch and, PostgreSQL 11-beta3

Hi. If "data-at-rest-encryption-wip-2018.07.25.patch" is applied to PostgreSQL 11-beta3 released last week, patch execution will fail as follows. patching file src/backend/replication/logical/reorderbuffer.c Hunk #9 FAILED at 2464. 1 out of 7 hunks FAILED -- saving rejects to

Re: "WIP: Data at rest encryption" patch and, 2 phase commit.

Hi. Applying https://www.postgresql.org/message-id/11678.1532519255%40localhost patch, the problem of pg_create_logical_replication_slot () and the 2PC problem were solved. Thanks. Antonin Houska wrote: > Toshi Harada wrote: > > > Hi. > > > > I applied the

"WIP: Data at rest encryption" patch and, 2 phase commit.

Hi. I applied the patch "WIP: Data at rest encryption" to PostgreSQL 11 - beta 2 and I'm working on it. When this patch is applied, the following problem occurs. * An error occurs when CHECKPOINT is executed during two-phase commit. * After an error occurs, if you stop

Re: "WIP: Data at rest encryption" patch and, 2 phase commit.

Toshi Harada wrote: > Hi. > > I applied the patch "WIP: Data at rest encryption" to PostgreSQL 11 - beta 2 > and I'm working on it. > > When this patch is applied, the following problem occurs. > > * An error occurs when CHECKPOINT is executed during tw

Re: [HACKERS] WIP: Data at rest encryption

Hi. I am interested in a patch of "WIP: Data at rest encryption". This patch("data-at-rest-encryption-wip-2018.06.27.patch") is applied to PostgreSQL 11-beta 2 and it is running. In the explanation of this patch, since "data stored during logical decoding"