Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Aug 7, 2019 at 2:55 AM Bruce Momjian wrote: > > On Wed, Aug 7, 2019 at 12:31:58AM +0900, Masahiko Sawada wrote: > > Well, so you mean that for example we encrypt only 100 bytes WAL > > record when append 100 bytes WAL records? > > > > For WAL encryption, if we encrypt the entire 8k WAL pa

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Aug 6, 2019 at 06:13:30PM -0400, Jonathan Katz wrote: > Hi, > > On 8/6/19 3:01 PM, Bruce Momjian wrote: > > On Tue, Aug 6, 2019 at 01:55:38PM -0400, Bruce Momjian wrote: > >> CTR mode creates a bit stream for the first 16 bytes with nonce of > >> (segment_number, counter = 0), and the ne

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Hi, On 8/6/19 3:01 PM, Bruce Momjian wrote: > On Tue, Aug 6, 2019 at 01:55:38PM -0400, Bruce Momjian wrote: >> CTR mode creates a bit stream for the first 16 bytes with nonce of >> (segment_number, counter = 0), and the next 16 bytes with >> (segment_number, counter = 1), etc. We only XOR using

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Aug 6, 2019 at 01:55:38PM -0400, Bruce Momjian wrote: > CTR mode creates a bit stream for the first 16 bytes with nonce of > (segment_number, counter = 0), and the next 16 bytes with > (segment_number, counter = 1), etc. We only XOR using the parts of the > bit stream we want to use. We

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Aug 7, 2019 at 12:31:58AM +0900, Masahiko Sawada wrote: > Well, so you mean that for example we encrypt only 100 bytes WAL > record when append 100 bytes WAL records? > > For WAL encryption, if we encrypt the entire 8k WAL page and write the > entire page, the encrypted-and-written page w

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Hi, On Wed, Aug 7, 2019, 00:31 Masahiko Sawada wrote: > Hi Bruce, > (off-list) > > I think I'm missing something about basic of encryption. Please let me > question about it on off-list. > Sorry for the noise, it was not off-list. I made a mistake. > On Tue, Aug 6, 2019 at 11:36 PM Bruce Momj

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Hi Bruce, (off-list) I think I'm missing something about basic of encryption. Please let me question about it on off-list. On Tue, Aug 6, 2019 at 11:36 PM Bruce Momjian wrote: > > On Tue, Aug 6, 2019 at 12:00:27PM +0900, Masahiko Sawada wrote: > > What I'm thinking about WAL encryption is that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Aug 6, 2019 at 12:00:27PM +0900, Masahiko Sawada wrote: > What I'm thinking about WAL encryption is that WAL records on WAL > buffer is not encrypted. When writing to the disk we copy the contents > of 8k WAL page to a temporary buffer and encrypt it, and then write > it. And according to

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Aug 6, 2019 at 9:42 AM Bruce Momjian wrote: > > On Wed, Jul 31, 2019 at 04:58:49PM +0900, Masahiko Sawada wrote: > > On Wed, Jul 31, 2019 at 3:29 PM Masahiko Sawada > > wrote: > > > > > > > > > For WAL encryption, before flushing WAL we encrypt whole 8k WAL page > > > and then write onl

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 09:25:01AM -0400, Sehrope Sarkuni wrote: > On Tue, Jul 30, 2019 at 4:48 PM Bruce Momjian wrote: > > I had more time to think about the complexity of adding relfilenode to > the IV.  Since relfilenode is only unique within a database/tablespace, > we would need

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 04:11:03PM +0900, Masahiko Sawada wrote: > On Wed, Jul 31, 2019 at 5:48 AM Bruce Momjian wrote: > > I am thinking for the heap/index IV, it would be: > > > > uint64 lsn; > > unint32 page number; > > /* only uses 11 bits for a zero-based CTR counter f

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 09:43:00AM -0400, Sehrope Sarkuni wrote: > On Wed, Jul 31, 2019 at 2:32 AM Masahiko Sawada wrote: > > Just to confirm, we have 21 bits left for nonce in CTR? We have LSN (8 > bytes), page-number (4 bytes) and counter (11 bits) in 16 bytes nonce > space. Even th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 04:58:49PM +0900, Masahiko Sawada wrote: > On Wed, Jul 31, 2019 at 3:29 PM Masahiko Sawada wrote: > > > > > > For WAL encryption, before flushing WAL we encrypt whole 8k WAL page > > and then write only the encrypted data of the new WAL record using > > pg_pwrite() rather

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 03:29:59PM +0900, Masahiko Sawada wrote: > Just to confirm, we have 21 bits left for nonce in CTR? We have LSN (8 > bytes), page-number (4 bytes) and counter (11 bits) in 16 bytes nonce > space. Even though we have 21 bits left we cannot store relfilenode to > the IV. No.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 2:32 AM Masahiko Sawada wrote: > Just to confirm, we have 21 bits left for nonce in CTR? We have LSN (8 > bytes), page-number (4 bytes) and counter (11 bits) in 16 bytes nonce > space. Even though we have 21 bits left we cannot store relfilenode to > the IV. > Fields like

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 4:48 PM Bruce Momjian wrote: > I had more time to think about the complexity of adding relfilenode to > the IV. Since relfilenode is only unique within a database/tablespace, > we would need to have pg_upgrade preserve database/tablespace oids > (which I assume are the sa

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 3:29 PM Masahiko Sawada wrote: > > > For WAL encryption, before flushing WAL we encrypt whole 8k WAL page > and then write only the encrypted data of the new WAL record using > pg_pwrite() rather than write whole encrypted page. So each time we > encrypt 8k WAL page we end

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 10:45 PM Sehrope Sarkuni wrote: > > On Tue, Jul 30, 2019 at 8:16 AM Masahiko Sawada wrote: >> >> On Mon, Jul 29, 2019 at 8:18 PM Sehrope Sarkuni wrote: >> > >> > On Mon, Jul 29, 2019 at 6:42 AM Masahiko Sawada >> > wrote: >> > > > An argument could be made to push that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Wed, Jul 31, 2019 at 5:48 AM Bruce Momjian wrote: > > On Tue, Jul 30, 2019 at 10:14:14AM -0400, Sehrope Sarkuni wrote: > > > In general it's fine to use the same IV with different keys. Only > > reuse > > of Key > > > + IV is a problem and the entire set of possible counter values

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 10:44 PM Bruce Momjian wrote: > > On Mon, Jul 29, 2019 at 08:43:06PM +0900, Masahiko Sawada wrote: > > > I am thinking of writing some Assert() code that checks that all buffers > > > using a single LSN are from the same relation (and therefore different > > > page numbers)

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 10:14:14AM -0400, Sehrope Sarkuni wrote: > > In general it's fine to use the same IV with different keys. Only reuse > of Key > > + IV is a problem and the entire set of possible counter values (IV + 0, > IV + > > 1, ...) generated with a key must be uniq

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 10:06 AM Masahiko Sawada wrote: > On Tue, Jul 30, 2019 at 5:03 AM Sehrope Sarkuni > wrote: > > > > On Mon, Jul 29, 2019 at 9:44 AM Bruce Momjian wrote: > >> > >> > Checking that all buffers using a single LSN are from the same > >> > relation would be a good idea but I t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 8:16 AM Bruce Momjian wrote: > On Tue, Jul 30, 2019 at 07:44:20AM -0400, Sehrope Sarkuni wrote: > > If each relation file has its own derived key, the derived TDEK for that > > relation file, then there is no issue with reusing the same IV = LSN || > Page > > Number. The T

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 5:03 AM Sehrope Sarkuni wrote: > > On Mon, Jul 29, 2019 at 9:44 AM Bruce Momjian wrote: >> >> > Checking that all buffers using a single LSN are from the same >> > relation would be a good idea but I think it's hard to test it and >> > regard the test result as okay. Even

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 8:16 AM Masahiko Sawada wrote: > On Mon, Jul 29, 2019 at 8:18 PM Sehrope Sarkuni > wrote: > > > > On Mon, Jul 29, 2019 at 6:42 AM Masahiko Sawada > wrote: > > > > An argument could be made to push that problem upstream, i.e. let the > > > > supplier of the passphrase dea

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 30, 2019 at 07:44:20AM -0400, Sehrope Sarkuni wrote: > On Mon, Jul 29, 2019 at 8:35 PM Bruce Momjian wrote: > From the patch: > > /* > ! * The initialization vector (IV) is used for page-level > ! * encryption.  We use the LSN and page number as the IV, and IV > ! * values must never

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 8:18 PM Sehrope Sarkuni wrote: > > On Mon, Jul 29, 2019 at 6:42 AM Masahiko Sawada wrote: > > > An argument could be made to push that problem upstream, i.e. let the > > > supplier of the passphrase deal with the indirection. You would still > > > need to verify the suppli

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 8:35 PM Bruce Momjian wrote: > On Sun, Jul 28, 2019 at 10:33:03PM -0400, Bruce Momjian wrote: > > I am thinking of writing some Assert() code that checks that all buffers > > using a single LSN are from the same relation (and therefore different > > page numbers). I would

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sun, Jul 28, 2019 at 10:33:03PM -0400, Bruce Momjian wrote: > I am thinking of writing some Assert() code that checks that all buffers > using a single LSN are from the same relation (and therefore different > page numbers). I would do it by creating a static array, clearing it on > XLogBeginIn

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 05:53:40PM -0400, Sehrope Sarkuni wrote: > I don't think this will be an issue in practice, but it should be documented. > Otherwise, it's not unreasonable for someone to expect that a promoted replica > would use be using new keys for everything after each promotion. > > E

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 04:09:52PM -0400, Alvaro Herrera wrote: > On 2019-Jul-27, Bruce Momjian wrote: > > > I think using LSN and page number, we will _never_ reuse the IV, except > > for cases like promoting two standbys, which I think we have to document > > as an insecure practice. > > Actual

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 7/29/19 6:11 PM, Sehrope Sarkuni wrote: > On Mon, Jul 29, 2019 at 4:15 PM Alvaro Herrera > wrote: > > On 2019-Jul-27, Sehrope Sarkuni wrote: > > > Given the non-cryptographic nature of CRC and its 16-bit size, I'd > > round down the malicious tampe

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 4:15 PM Alvaro Herrera wrote: > On 2019-Jul-27, Sehrope Sarkuni wrote: > > > Given the non-cryptographic nature of CRC and its 16-bit size, I'd > > round down the malicious tamper detection it provides to zero. At best > > it catches random disk errors so might as well kee

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 4:10 PM Alvaro Herrera wrote: > On 2019-Jul-27, Bruce Momjian wrote: > > > I think using LSN and page number, we will _never_ reuse the IV, except > > for cases like promoting two standbys, which I think we have to document > > as an insecure practice. > > Actually, why is

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 2019-Jul-27, Sehrope Sarkuni wrote: > Given the non-cryptographic nature of CRC and its 16-bit size, I'd > round down the malicious tamper detection it provides to zero. At best > it catches random disk errors so might as well keep it in plain text > and checkable offline. But what attack are

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 2019-Jul-27, Bruce Momjian wrote: > I think using LSN and page number, we will _never_ reuse the IV, except > for cases like promoting two standbys, which I think we have to document > as an insecure practice. Actually, why is it an insecure practice? If you promote two standbys, then the enc

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 9:44 AM Bruce Momjian wrote: > > Checking that all buffers using a single LSN are from the same > > relation would be a good idea but I think it's hard to test it and > > regard the test result as okay. Even if we passed 'make checkworld', > > it might still be possible to

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sun, Jul 28, 2019 at 10:33:03PM -0400, Bruce Momjian wrote: > I did some more research on which cases use a single LSN to modify > multiple 8k pages. The normal program flow is: > > XLogBeginInsert(); > ... > --> XLogRegisterBuffer(0, meta, ... > recptr = XLogInsert(R

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 08:43:06PM +0900, Masahiko Sawada wrote: > > I am thinking of writing some Assert() code that checks that all buffers > > using a single LSN are from the same relation (and therefore different > > page numbers). I would do it by creating a static array, clearing it on > > X

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019, 20:43 Masahiko Sawada wrote: > That way, we never reuse IV in a different relation with the same page > number because relNode is unique within a database in a particular > tablespace as you mentioned. > Sorry, I meant that we can ensure IV+key is unique. -- Maaahiko Sawad

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 11:33 AM Bruce Momjian wrote: > > On Thu, Jul 25, 2019 at 01:03:06PM -0400, Bruce Momjian wrote: > > On Tue, Jul 16, 2019 at 01:24:54PM +0900, Masahiko Sawada wrote: > > > On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > > > then each row change gets its own LSN.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 6:42 AM Masahiko Sawada wrote: > > An argument could be made to push that problem upstream, i.e. let the > > supplier of the passphrase deal with the indirection. You would still > > need to verify the supplied passphrase/key is correct via something > > like authenticating

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 7:17 PM Sehrope Sarkuni wrote: > > On Mon, Jul 29, 2019 at 4:39 AM Masahiko Sawada wrote: > > After more thoughts, I'm confused why we need to have MDEK. We can use > > KEK derived from passphrase and TDEK and WDEK that are derived from > > KEK. That way, we don't need sto

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 29, 2019 at 4:39 AM Masahiko Sawada wrote: > After more thoughts, I'm confused why we need to have MDEK. We can use > KEK derived from passphrase and TDEK and WDEK that are derived from > KEK. That way, we don't need store any key in database file. What is > the advantage of 3-tier key

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sat, Jul 27, 2019 at 12:27 AM Masahiko Sawada wrote: > > On Fri, Jul 26, 2019 at 10:57 AM Jonathan S. Katz > wrote: > > > > Hi, > > > > Before my reply, I wanted to say that I've been lurking on this thread > > for a bit as I've tried to better inform myself on encryption at rest > > and how

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 01:03:06PM -0400, Bruce Momjian wrote: > On Tue, Jul 16, 2019 at 01:24:54PM +0900, Masahiko Sawada wrote: > > On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > > then each row change gets its own LSN. You are asking if an update that > > > just expires one row and

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sat, Jul 27, 2019 at 03:02:02PM -0400, Sehrope Sarkuni wrote: > On Sat, Jul 27, 2019 at 1:32 PM Bruce Momjian wrote: > > Uh, I listed the three options for the CRC and gave the benefits of > > each: > > > > > > https://www.postgresql.org/message-id/20190725200343.xo4dcjm5azrfn...@momji

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 7/27/19 3:02 PM, Sehrope Sarkuni wrote: > More generally, without a cryptographic MAC I don't think it's > possible to provide any meaningful malicious tamper detection. And > even that would have to be off-page to deal with page replay (which I > think is out of scope). > > [1]: https://en.wik

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sat, Jul 27, 2019 at 1:32 PM Bruce Momjian wrote: > Uh, I listed the three options for the CRC and gave the benefits of > each: > > > https://www.postgresql.org/message-id/20190725200343.xo4dcjm5azrfn...@momjian.us > > Obviously I was not clear on the benefits. To quote: > > 1

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 11:30:55PM -0400, Alvaro Herrera wrote: > On 2019-Jul-25, Alvaro Herrera wrote: > > > > Uh, there are no known attacks on AES with known plain-text, e.g., SSL > > > uses AES, so I think we are good with encrypting everything after the > > > first 16 bytes. > > > > Well, ma

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 10:57:08PM -0400, Alvaro Herrera wrote: > On 2019-Jul-25, Bruce Momjian wrote: > > > On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > > > > Why are we encrypting the page header in the first place? It seems to > > > me that the encrypted area should cover

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 26, 2019 at 10:57 AM Jonathan S. Katz wrote: > > Hi, > > Before my reply, I wanted to say that I've been lurking on this thread > for a bit as I've tried to better inform myself on encryption at rest > and how it will apply to what we want to build. I actually built a > (poor) prototyp

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 2019-Jul-25, Alvaro Herrera wrote: > > Uh, there are no known attacks on AES with known plain-text, e.g., SSL > > uses AES, so I think we are good with encrypting everything after the > > first 16 bytes. > > Well, maybe there aren't any attacks *now*, but I don't know what will > happen in the

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 2019-Jul-25, Bruce Momjian wrote: > On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > > Why are we encrypting the page header in the first place? It seems to > > me that the encrypted area should cover only the line pointers and the > > tuple data area; the page header needs t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Hi, Before my reply, I wanted to say that I've been lurking on this thread for a bit as I've tried to better inform myself on encryption at rest and how it will apply to what we want to build. I actually built a (poor) prototype in Python of the key management system that Joe & Masahiko both laid

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Buffer Encryption == We will use AES-CBC for buffer encryption. We will add key id (4byte) >>> >>> I think we might want to use CTR for this, and will post after this. > > Not sure if I missed this post or not (as several people mentioned, it > is easy to get lost in th

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 09:11:18PM -0400, Sehrope Sarkuni wrote: > On Thu, Jul 25, 2019 at 8:50 PM Bruce Momjian wrote: > > On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > > You can still use CTR mode and include those to make the key + IV unique > by > > adding

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 8:50 PM Bruce Momjian wrote: > On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > > You can still use CTR mode and include those to make the key + IV unique > by > > adding them to the derived key rather than the IV. > > > > The IV per-page would still be L

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 08:44:40PM -0400, Sehrope Sarkuni wrote: > On Thu, Jul 25, 2019 at 7:51 PM Bruce Momjian wrote: > > Looking at the bits we have, the IV for AES is 16 bytes.  Since we know > we have to use LSN (to change the IV for each page write), and the page > number (so WA

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 08:07:28PM -0400, Stephen Frost wrote: > > Yes, we need to see how we are going to do that for checksums and > > encryption and come up with a plan. > > This is already being done though- Andres has a patch posted already and > my recollection from a quick look at that is t

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 7:51 PM Bruce Momjian wrote: > Looking at the bits we have, the IV for AES is 16 bytes. Since we know > we have to use LSN (to change the IV for each page write), and the page > number (so WAL updates that change multiple pages with the same LSN use > different IVs), that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 07:41:14PM -0400, Stephen Frost wrote: > > > You are right that we can allow it online, but we haven't been > > > discussing these cases since it is easy to do this because we have > > > access to the keys. I do think

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 07:41:14PM -0400, Stephen Frost wrote: > > You are right that we can allow it online, but we haven't been > > discussing these cases since it is easy to do this because we have > > access to the keys. I do think whatever code we use for checksum online > > changes will be u

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote: > Masahiko Sawada copied this section as a desired direction, so I want to > drill down into it. I think we have five possible approaches for level > 3 listed above. > > The simplest approach would be to say that the LSN/page-number a

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 05:50:57PM -0400, Stephen Frost wrote: > > > > > pg_upgrade seems immune to must of this, and that is by design. > > > > > However, I am hesitant to change the heap/index page format for > > > > > encryption because if

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 05:50:57PM -0400, Stephen Frost wrote: > > > > pg_upgrade seems immune to must of this, and that is by design. > > > > However, I am hesitant to change the heap/index page format for > > > > encryption because if we add fields, old pages might not fit as > > > > encrypted p

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 03:55:01PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > > After

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 15, 2019 at 06:08:28PM -0400, Sehrope Sarkuni wrote: > Hi, > > Some more thoughts on CBC vs CTR modes. There are a number of > advantages to using CTR mode for page encryption. > > CTR encryption modes can be fully parallelized, whereas CBC can only > parallelized for decryption. Whil

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 03:55:01PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > > > Greetings, > > > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > After talking to Joe Conway,

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 03:43:34PM -0400, Alvaro Herrera wrote: > On 2019-Jul-15, Bruce Momjian wrote: > > > Uh, if someone modifies a few bytes of the page, we will decrypt it, but > > the checksum (per-page or WAL) will not match our decrypted output. How > > would they make it match the checks

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > > Greetings, > > > > * Bruce Momjian (br...@momjian.us) wrote: > > > After talking to Joe Conway, I just want to mention that if we decide > > > that the LSN is unique among hea

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 03:41:05PM -0400, Stephen Frost wrote: > Greetings, > > * Bruce Momjian (br...@momjian.us) wrote: > > After talking to Joe Conway, I just want to mention that if we decide > > that the LSN is unique among heap and index, or among heap or index, we > > will need to make sure

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 2019-Jul-15, Bruce Momjian wrote: > Uh, if someone modifies a few bytes of the page, we will decrypt it, but > the checksum (per-page or WAL) will not match our decrypted output. How > would they make it match the checksum without already knowing the key. > I read [1] but could not see that e

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Greetings, * Bruce Momjian (br...@momjian.us) wrote: > After talking to Joe Conway, I just want to mention that if we decide > that the LSN is unique among heap and index, or among heap or index, we > will need to make sure future WAL records retain this uniqueness. One thing comes to mind regard

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 02:05:05PM -0400, Bruce Momjian wrote: > The second approach is to say they will collide and that we need to mix > a constant into the IV for tables/indexes and a different one for WAL. > In a way I would like to mix the pg_controldata Database system > Identifier into ther

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 19, 2019 at 01:59:41PM +0200, Tomas Vondra wrote: > On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote: > > We can guarantee integrity and authenticity of backup, but that's a separate > > feature: someone may need this although it's o.k. for him to run the cluster > > unenc

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sat, Jul 20, 2019 at 07:30:30PM +0200, Tomas Vondra wrote: > Forbid checksums? I don't see how that could be acceptable. We either have > to accept the limitations of the current design (having to decrypt > everything before checking the checksums) or change the design. Yes, checksums certainly

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 26, 2019 at 02:54:04AM +0900, Masahiko Sawada wrote: > On Fri, Jul 26, 2019 at 2:18 AM Bruce Momjian wrote: > > > > On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote: > > > I've re-considered the design of TDE feature based on the discussion > > > so far. The one of the m

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sun, Jul 14, 2019 at 12:13:45PM -0400, Joe Conway wrote: > In my email I linked the wrong page for [2]. The correct one is here: > [2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html > > Following that, I think we could end up with three tiers: > > 1. A master key encryption ke

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 26, 2019 at 2:18 AM Bruce Momjian wrote: > > On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote: > > I've re-considered the design of TDE feature based on the discussion > > so far. The one of the main open question is the granular of > > encryption objects: cluster encryp

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 25, 2019 at 01:18:44PM -0400, Bruce Momjian wrote: > > Key Management > > == > > We will use 3-tier key architecture as Joe proposed. > > > > 1. A master key encryption key (KEK): this is the ley supplied by the > > database admin using something akin to ssl_passphra

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Thu, Jul 18, 2019 at 12:04:25PM +0900, Masahiko Sawada wrote: > I've re-considered the design of TDE feature based on the discussion > so far. The one of the main open question is the granular of > encryption objects: cluster encryption or more-granular-than-cluster > encryption. The followings

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 16, 2019 at 01:24:54PM +0900, Masahiko Sawada wrote: > On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > then each row change gets its own LSN. You are asking if an update that > > just expires one row and adds it to a new page gets the same LSN. I > > don't know. > > The fo

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sat, Jul 20, 2019 at 03:39:25PM -0400, Sehrope Sarkuni wrote: > How about storing the CRC of the encrypted pages? It would not leak > any additional information and serves the same purpose as a > non-encrypted one, namely I/O corruption detection. I took a look at > pg_checksum and besides check

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 15, 2019 at 07:39:20PM -0400, Alvaro Herrera wrote: > On 2019-Jul-15, Bruce Momjian wrote: > > > My point is that doing encryption of only some data might actually make > > the system slower due to the lookups, so I think we need to implement > > all-cluster encryption and then see wha

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sat, Jul 20, 2019 at 1:30 PM Tomas Vondra wrote: > Forbid checksums? I don't see how that could be acceptable. We either have > to accept the limitations of the current design (having to decrypt > everything before checking the checksums) or change the design. > > I personally think we should d

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 19, 2019 at 04:02:19PM +0200, Antonin Houska wrote: Tomas Vondra wrote: On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote: >Tomas Vondra wrote: > >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: >> >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vond

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Tomas Vondra wrote: > On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote: > >Tomas Vondra wrote: > > > >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: > >> >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > >> >> One extra thing we should consider is a

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 19, 2019 at 01:32:01PM +0200, Antonin Houska wrote: Tomas Vondra wrote: On Mon, Jul 15, 2019 at 06:11:41PM -0400, Bruce Momjian wrote: >On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote: >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: >> > On Sat, Jul 1

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 19, 2019 at 12:04:36PM +0200, Antonin Houska wrote: Tomas Vondra wrote: On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: >> One extra thing we should consider is authenticated encryption. We can't >> just

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Tomas Vondra wrote: > On Mon, Jul 15, 2019 at 06:11:41PM -0400, Bruce Momjian wrote: > >On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote: > >> On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: > >> > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > >> > > O

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Tomas Vondra wrote: > On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: > >On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > >> One extra thing we should consider is authenticated encryption. We can't > >> just encrypt the pages (no matter which AES mode is used - XTS/C

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Fri, Jul 12, 2019 at 7:37 AM Bruce Momjian wrote: > > On Wed, Jul 10, 2019 at 12:26:24PM -0400, Bruce Momjian wrote: > > On Wed, Jul 10, 2019 at 08:31:17AM -0400, Joe Conway wrote: > > > Please see my other reply (and > > > https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-3

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On 2019-Jul-15, Bruce Momjian wrote: > My point is that doing encryption of only some data might actually make > the system slower due to the lookups, so I think we need to implement > all-cluster encryption and then see what the overhead is, and if there > are use-cases for not encrypting only so

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 15, 2019 at 9:38 PM Antonin Houska wrote: > > Masahiko Sawada wrote: > > > On Mon, Jun 17, 2019 at 11:02 PM Tomas Vondra > > wrote: > > > > > > On Mon, Jun 17, 2019 at 08:39:27AM -0400, Joe Conway wrote: > > > >On 6/17/19 8:29 AM, Masahiko Sawada wrote: > > > >> From perspective of

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Sat, Jul 13, 2019 at 12:33 AM Bruce Momjian wrote: > > On Fri, Jul 12, 2019 at 02:15:02PM +0900, Masahiko Sawada wrote: > > > We will use CBC AES128 mode for tables/indexes, and CTR AES128 for WAL. > > > 8k pages will use the LSN as a nonce, which will be encrypted to > > > generate the initial

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 15, 2019 at 06:08:28PM -0400, Sehrope Sarkuni wrote: > Hi, > > Some more thoughts on CBC vs CTR modes. There are a number of > advantages to using CTR mode for page encryption. > > CTR encryption modes can be fully parallelized, whereas CBC can only > parallelized for decryption. Whil

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Tue, Jul 16, 2019 at 02:04:58AM +0200, Tomas Vondra wrote: > On Mon, Jul 15, 2019 at 06:05:37PM -0400, Bruce Momjian wrote: > > On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote: > > > On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote: > > > > The crazy seems more sane now

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 15, 2019 at 06:05:37PM -0400, Bruce Momjian wrote: On Mon, Jul 15, 2019 at 10:44:34PM +0200, Tomas Vondra wrote: On Mon, Jul 15, 2019 at 03:55:38PM -0400, Bruce Momjian wrote: > The crazy seems more sane now --- "encrypt the page with CRC contents as > zero" (which we probably alread

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

On Mon, Jul 15, 2019 at 06:11:41PM -0400, Bruce Momjian wrote: On Mon, Jul 15, 2019 at 11:05:30PM +0200, Tomas Vondra wrote: On Mon, Jul 15, 2019 at 03:42:39PM -0400, Bruce Momjian wrote: > On Sat, Jul 13, 2019 at 11:58:02PM +0200, Tomas Vondra wrote: > > One extra thing we should consider is au

<    1   2   3   4   5   >