hi, On Sun, Jan 25, 2015 at 11:06 AM, Jeff Holoman <jholo...@cloudera.com> wrote: > Justin, > > I'd be interested to learn specifically what security features you are > interested in.
we are interested in both simple authentication and per-topic authorization. we can get started with simple authentication, but will need per-topic authorization or encrypted message bodies before we can send some event data over Kafka. > There is a doc on the wiki here > https://cwiki.apache.org/confluence/display/KAFKA/Security > > As well as the Umbrella JIRA Here: > https://issues.apache.org/jira/browse/KAFKA-1682 thanks, i've read the security page in the wiki, and i'm following the jira issues. > Most security-related JIRAs have a component listed as security > https://issues.apache.org/jira/browse/KAFKA-1882?jql=project%20%3D%20KAFKA%20AND%20component%20%3D%20security > > There is a quite a bit of plumbing work to be done in order for all of the > security features to come together. There technically isn't a 0.9 branch at > the moment, all of the development is being applied to trunk and since > there isn't as of yet even a 0.8.3 branch, I don't think you'd be in a good > spot trying to pull in a bunch of stuff that isn't complete. right, our initial PoC work and planning for 0.8.2 and security has focused on simple wrappers around Kafka 0.8.2. for example, adding an iptables/security group whitelist. this only gets us part of the way, as by necessity we run customer code on a large minority of our productions machines (we host a large number of high traffic Drupal sites), and that code can legitimately talk to the internet. those instances should be significant producers connected to Kafka, but we can't simply trust traffic from those instances. we tested haproxy in front of Kafka, requiring a client SSL certificate for any code that talks to haproxy. then, we used advertised.port and advertised.hostname, and monkey-patched the posseidon ruby client to speak SSL. this works up to a point, but has it's own problems. if we use one Kafka cluster, we have to teach all clients to speak SSL and offer a valid certificate. or, we teach some clients to ignore advertised.hostname, and some to honour it, depending which method we use to trust the traffic. another option is to mirror traffic from the SSL-wrapped kafka to the iptables-wrapped Kafka. i also wrote a simple websocket <-> Kafka bridge using the Shopify sarama client, and that works nicely but would take some effort to bring to production readiness, and doesn't have any per-topic authorization. we think we can use a some combination of the above to start using Kafka for a large chunk of our log pipeline needs, but it's far from ideal compared to simply using built-in security features within Kafka. how do others solve these sorts of problems with Kafka? cheers Justin > Just my 2 cents. > > Thanks > > Jeff > > > On Sun, Jan 25, 2015 at 10:33 AM, Justin Randell <justin.rand...@acquia.com> > wrote: > >> Hi, >> >> We're assessing Kafka for use at Acquia. We run tens of thousands of >> customer websites and many internal applications on ~9k AWS EC2 >> instances. >> >> We're currently weighing the pros and cons of starting with 0.82 plus >> custom security, or waiting until the security features land in 0.9. >> >> How likely is Kafka 0.9 to ship in April 2015? (As seen here - >> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan.) >> >> How stable is the 0.9 branch? Is it crazy to consider running a 0.9 >> beta in production? >> >> Are there any existing patch sets against 0.8x that implement security >> features? >> >> thanks, >> Justin >> > > > > -- > Jeff Holoman > Systems Engineer
