I think that Hadoop and Cassandra do [1] (Transparent Encryption) We're doing [2] (on a side note, for [2] you still need authentication on the producer side - you don't want an unauthorized user writing garbage). Right now we have the 'user' doing the encryption and submitting raw bytes to the producer. I was suggesting implementing an encryptor in the producer itself - I think it's cleaner and can be reused by other users (instead of having to do their own encryption)
Cheers, Eugene On Fri, Jul 31, 2015 at 4:04 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > I think the goal here is to make the actual message stored on broker to be > encrypted, because after we have SSL, the transmission would be encrypted. > > In general there might be tow approaches: > 1. Broker do the encryption/decryption > 2. Client do the encryption/decryption > > From performance point of view, I would prefer [2]. It is just in that > case, maybe user does not necessarily need to use SSL anymore because the > data would be encrypted anyway. > > If we let client do the encryption, there are also two ways to do so - > either we let producer take an encryptor or users can do > serialization/encryption outside the producer and send raw bytes. The only > difference between the two might be flexibility. For example, if someone > wants to know the actual bytes of a message that got sent over the wire, > doing it outside the producer would probably more preferable. > > Jiangjie (Becket) Qin > > On Thu, Jul 30, 2015 at 12:16 PM, eugene miretsky < > eugene.miret...@gmail.com > > wrote: > > > Hi, > > > > Based on the security wiki page > > <https://cwiki.apache.org/confluence/display/KAFKA/Security> encryption > of > > data at rest is out of scope for the time being. However, we are > > implementing encryption in Kafka and would like to see if there is > > interest in submitting a patch got it. > > > > I suppose that one way to implement encryption would be to add an > > 'encrypted key' field to the Message/MessageSet structures in the > > wire protocole - however, this is a very big and fundamental change. > > > > A simpler way to add encryption support would be: > > 1) Custom Serializer, but it wouldn't be compatible with other custom > > serializers (Avro, etc. ) > > 2) Add a step in KafkaProducer after serialization to encrypt the data > > before it's being submitted to the accumulator (encryption is done in the > > submitting thread, not in the producer io thread) > > > > Is there interest in adding #2 to Kafka? > > > > Cheers, > > Eugene > > >
