Re: Tips for working with Kafka and data streams

2015-02-25 Thread Christian Csar
Yeah, we do have scenarios where we use customer specific keys so our envelopes end up containing key identification information for accessing our key repository. I'll certainly follow any changes you propose in this area with interest, but I'd expect that sort of centralized key thing to be fairly

Re: Tips for working with Kafka and data streams

2015-02-25 Thread Julio Castillo
Although full disk encryption appears to be an easy solution, in our case that may not be sufficient. For cases where the actual payload needs to be encrypted, the cost of encryption is paid by the consumer and producers. Further complicating the matter would be the handling of encryption keys, etc

Re: Tips for working with Kafka and data streams

2015-02-25 Thread Christian Csar
The questions we get from customers typically end up being general so we break out our answer into network level and on disk scenarios. On disk/at rest scenario may just be use full disk encryption at the OS level and Kafka doesn't need to worry about it. But documenting any issues around it would

Re: Tips for working with Kafka and data streams

2015-02-25 Thread Tong Li
nity Development Building 501/B205 liton...@us.ibm.com From: Jay Kreps To: "d...@kafka.apache.org" , "users@kafka.apache.org" Date: 02/25/2015 02:52 PM Subject: Tips for working with Kafka and data streams Hey guys, One thing we tried to do along

Re: Tips for working with Kafka and data streams

2015-02-25 Thread Jay Kreps
Hey Christian, That makes sense. I agree that would be a good area to dive into. Are you primarily interested in network level security or encryption on disk? -Jay On Wed, Feb 25, 2015 at 1:38 PM, Christian Csar wrote: > I wouldn't say no to some discussion of encryption. We're running on Azur

Re: Tips for working with Kafka and data streams

2015-02-25 Thread Christian Csar
I wouldn't say no to some discussion of encryption. We're running on Azure EventHubs (with preparations for Kinesis for EC2, and Kafka for deployments in customer datacenters when needed) so can't just use disk level encryption (which would have its own overhead). We're putting all of our messages

Tips for working with Kafka and data streams

2015-02-25 Thread Jay Kreps
Hey guys, One thing we tried to do along with the product release was start to put together a practical guide for using Kafka. I wrote this up here: http://blog.confluent.io/2015/02/25/stream-data-platform-1/ I'd like to keep expanding on this as good practices emerge and we learn more stuff. So