For 3, Confluent wrote a REST proxy that's pretty comprehensive. See the docs: http://confluent.io/docs/current/kafka-rest/docs/intro.html and a blog post describing it + future directions: http://blog.confluent.io/2015/03/25/a-comprehensive-open-source-rest-proxy-for-kafka/
There are a few other REST proxies: https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-HTTPREST But I don't think any of them support everything you need yet -- specifically, security stuff isn't included in any of them yet. You could address this with some performance hit by putting another HTTP server like nginx in front of the proxies (locally with each instance) to get that control. For Confluent's proxy, we're also thinking about how to add security features since we'll need something to protect admin operations: https://github.com/confluentinc/kafka-rest/wiki/Project---Admin-APIs -Ewen On Tue, Mar 24, 2015 at 11:34 PM, Nagesh <nageswara.r...@gmail.com> wrote: > Hi, > > I think for 2) you can use Kafka Consumer and push messages to vertex event > bus, which already have REST implementation (vertx-jersey). > > I would say, Vertx cluster can be used as receive data irrespective of > topic and then publish to particular kafka topic. Then consume messages > from kafka by different consumer and distribute. Kafka can hold messages > without dropping at bursts and even at the time down stream slows down. > > Regards, > Nageswara Rao > > On Wed, Mar 25, 2015 at 10:58 AM, Manoj Khangaonkar <khangaon...@gmail.com > > > wrote: > > > Hi, > > > > For (1) and perhaps even for (2) where distribution/filtering on scale is > > required, I would look at using Apache Storm with kafka. > > > > For (3) , it seems you just need REST services wrapping kafka > > consumers/producers. I would start with usual suspects like jersey. > > > > regards > > > > On Tue, Mar 24, 2015 at 12:06 PM, Valentin <kafka-9999...@sblk.de> > wrote: > > > > > > > > Hi guys, > > > > > > we have three Kafka use cases for which we have written our own PoC > > > implementations, > > > but where I am wondering whether there might be any fitting open source > > > solution/tool/framework out there. > > > Maybe someone of you has some ideas/pointers? :) > > > > > > 1) Message routing/distribution/filter tool > > > We need to copy messages from a set of input topics to a set of output > > > topics > > > based on their message key values. Each message in an input topic will > go > > > to 0 to N output topics, > > > each output topic will receive messages from 0 to N input topics. > > > So basically the tool acts as a message routing component in our > system. > > > Example configuration: > > > <input topic A>:<output topic K>:<key value 1>,<key value 2>,<key value > > 3> > > > <input topic A>:<output topic L>:<key value 2>,<key value 4> > > > <input topic B>:<output topic K>:<key value 5>,<key value 6> > > > ... > > > It would also be interesting to define distribution/filter rules based > on > > > regular expressions on the message key or message body. > > > > > > 2) Kafka-to-REST Push service > > > We need to consume messages from a set of topics, translate them into > > REST > > > web service calls > > > and forward the data to existing, non-Kafka-aware systems with REST > APIs > > > that way. > > > > > > 3) HTTP REST API for consumers and producers > > > We need to expose the simple consumer and the producer functionalities > > via > > > REST web service calls, > > > with authentication and per-topic-authorization on REST API level and > TLS > > > for transport encryption. > > > Offset tracking is done by the connected systems, not by the > > > broker/zookeeper/REST API. > > > We expect a high message volume in the future here, so performance > would > > > be a key concern. > > > > > > Greetings > > > Valentin > > > > > > > > > > > -- > > http://khangaonkar.blogspot.com/ > > > > > > -- > Thanks, > Nageswara Rao.V > > *"The LORD reigns"* > -- Thanks, Ewen
