Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi Andrew, thanks for your feedback! I am interested though, why are you doubtful about getting a committer to volunteer an opinion? Shouldn't this be in their interest as well? I'll just continue along for now and start building a very rough poc implementation based on what's in the KIP so far

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Thanks for your feedback both of you! I've commented inline below. On Thu, 8 Aug 2019 at 08:38, Jörn Franke wrote: > If you are doing batch encryption then you are more similar to a scenario > of file encryption. The more frequent the messages are you are closer to > the ssl/https scenarios.

Re: [DISCUSS] KIP-317: Transparent Data Encryption

If you are doing batch encryption then you are more similar to a scenario of file encryption. The more frequent the messages are you are closer to the ssl/https scenarios. You may learn from those protocols on how they handle keys, how long they keep them etc. to implement your E2e solution .

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi Sönke Liebau Thanks for the great detailed documentation. However, I feel by leaving the KMS outside of Kafka might simplify the whole thing to a great extent. If the broker is not going to touch the

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi, I think this is a useful KIP and it looks good in principle. While it can all be done using interceptors, if the brokers do not know anything about it, you need to maintain the mapping from topics to key ids somewhere external. I'd prefer the way you've done it. I'm not sure whether you'll

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi, I have so far received pretty much no comments on the technical details outlined in the KIP. While I am happy to continue with my own ideas of how to implement this, I would much prefer to at least get a very broad "looks good in principle, but still lots to flesh out" from a few people

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi everybody, I'd like to rekindle the discussion around KIP-317. I have reworked the KIP a little bit in order to design everything as a pluggable implementation. During the course of that work I've also decided to rename the KIP, as encryption will only be transparent in some cases. It is now

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi Viktor, thanks for your input! We could accommodate magic headers by removing any known fixed bytes pre-encryption, sticking them in a header field and prepending them after decryption. However, I am not sure whether this is actually necessary, as most modern (AES for sure) algorithms are

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi Sönke, Compressing before encrypting has its dangers as well. Suppose you have a known compression format which adds a magic header and you're using a block cipher with a small enough block, then it becomes much easier to figure out the encryption key. For instance you can look at Snappy's

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi Stephane, thanks for pointing out the broken pictures, I fixed those. Regarding encrypting before or after batching the messages, you are correct, I had not thought of compression and how this changes things. Encrypted data does not really encrypt well. My reasoning at the time of writing was

Re: [DISCUSS] KIP-317: Transparent Data Encryption

Hi Sonke Very much needed feature and discussion. FYI the image links seem broken. My 2 cents (if I understood correctly): you say "This process will be implemented after Serializer and Interceptors are done with the message right before it is added to the batch to be sent, in order to ensure

[DISCUSS] KIP-317: Transparent Data Encryption

Hi everybody, I've created a draft version of KIP-317 which describes the addition of transparent data encryption functionality to Kafka. Please consider this as a basis for discussion - I am aware that this is not at a level of detail sufficient for implementation, but I wanted to get some