Good points :) 1) Kafka already (pending commit) has an authorization layer, so theoretically we are good for SOX, HIPAA, PCI, etc. Transparent broker encryption will support PCI never-let-unencrypted-card-number-hit-disk.
2) Agree on Key Management being complete PITA. It may better to centralize this pain in the broker rather than distributing it to clients. Any reason you think its better to let the clients handle it? The way I see it, we'll need to handle key management the way we did authorization - give an API for interfacing with existing systems. More important, we need the broker to be able to decrypt and encrypt in order to support compaction (unless we can find a cool key-uniqueness-preserving encryption algorithm, but this may not be as secure). I think we also need the broker to be able to re-compress data, and since we always encrypt compressed bits (compressing encrypted bits doesn't compress), we need the broker to decrypt before re-compressing. On Fri, Jul 31, 2015 at 2:27 PM, Todd Palino <tpal...@gmail.com> wrote: > It does limit it to clients that have an implementation for encryption, > however encryption on the client side is better from an auditing point of > view (whether that is SOX, HIPAA, PCI, or something else). Most of those > types of standards are based around allowing visibility of data to just the > people who need it. That includes the admins of the system (who are often > not the people who use the data). > > Additionally, key management is a royal pain, and there are lots of > different types of systems that one may want to use. This is a pretty big > complication for the brokers. > > -Todd > > > On Fri, Jul 31, 2015 at 2:21 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> I've seen interest in HDFS-like "encryption zones" in Kafka. >> >> This has the advantage of magically encrypting data at rest regardless >> of which client is used as a producer. >> Adding it on the client side limits the feature to the java client. >> >> Gwen >> >> On Fri, Jul 31, 2015 at 1:20 PM, eugene miretsky >> <eugene.miret...@gmail.com> wrote: >> > I think that Hadoop and Cassandra do [1] (Transparent Encryption) >> > >> > We're doing [2] (on a side note, for [2] you still need authentication on >> > the producer side - you don't want an unauthorized user writing garbage). >> > Right now we have the 'user' doing the encryption and submitting raw >> bytes >> > to the producer. I was suggesting implementing an encryptor in the >> > producer itself - I think it's cleaner and can be reused by other users >> > (instead of having to do their own encryption) >> > >> > Cheers, >> > Eugene >> > >> > On Fri, Jul 31, 2015 at 4:04 PM, Jiangjie Qin <j...@linkedin.com.invalid >> > >> > wrote: >> > >> >> I think the goal here is to make the actual message stored on broker to >> be >> >> encrypted, because after we have SSL, the transmission would be >> encrypted. >> >> >> >> In general there might be tow approaches: >> >> 1. Broker do the encryption/decryption >> >> 2. Client do the encryption/decryption >> >> >> >> From performance point of view, I would prefer [2]. It is just in that >> >> case, maybe user does not necessarily need to use SSL anymore because >> the >> >> data would be encrypted anyway. >> >> >> >> If we let client do the encryption, there are also two ways to do so - >> >> either we let producer take an encryptor or users can do >> >> serialization/encryption outside the producer and send raw bytes. The >> only >> >> difference between the two might be flexibility. For example, if someone >> >> wants to know the actual bytes of a message that got sent over the wire, >> >> doing it outside the producer would probably more preferable. >> >> >> >> Jiangjie (Becket) Qin >> >> >> >> On Thu, Jul 30, 2015 at 12:16 PM, eugene miretsky < >> >> eugene.miret...@gmail.com >> >> > wrote: >> >> >> >> > Hi, >> >> > >> >> > Based on the security wiki page >> >> > <https://cwiki.apache.org/confluence/display/KAFKA/Security> >> encryption >> >> of >> >> > data at rest is out of scope for the time being. However, we are >> >> > implementing encryption in Kafka and would like to see if there is >> >> > interest in submitting a patch got it. >> >> > >> >> > I suppose that one way to implement encryption would be to add an >> >> > 'encrypted key' field to the Message/MessageSet structures in the >> >> > wire protocole - however, this is a very big and fundamental change. >> >> > >> >> > A simpler way to add encryption support would be: >> >> > 1) Custom Serializer, but it wouldn't be compatible with other custom >> >> > serializers (Avro, etc. ) >> >> > 2) Add a step in KafkaProducer after serialization to encrypt the >> data >> >> > before it's being submitted to the accumulator (encryption is done in >> the >> >> > submitting thread, not in the producer io thread) >> >> > >> >> > Is there interest in adding #2 to Kafka? >> >> > >> >> > Cheers, >> >> > Eugene >> >> > >> >> >>
