Re: Message routing, Kafka-to-REST and HTTP API tools/frameworks for Kafka?
Hi, I think for 2) you can use Kafka Consumer and push messages to vertex event bus, which already have REST implementation (vertx-jersey). I would say, Vertx cluster can be used as receive data irrespective of topic and then publish to particular kafka topic. Then consume messages from kafka by different consumer and distribute. Kafka can hold messages without dropping at bursts and even at the time down stream slows down. Regards, Nageswara Rao On Wed, Mar 25, 2015 at 10:58 AM, Manoj Khangaonkar khangaon...@gmail.com wrote: Hi, For (1) and perhaps even for (2) where distribution/filtering on scale is required, I would look at using Apache Storm with kafka. For (3) , it seems you just need REST services wrapping kafka consumers/producers. I would start with usual suspects like jersey. regards On Tue, Mar 24, 2015 at 12:06 PM, Valentin kafka-...@sblk.de wrote: Hi guys, we have three Kafka use cases for which we have written our own PoC implementations, but where I am wondering whether there might be any fitting open source solution/tool/framework out there. Maybe someone of you has some ideas/pointers? :) 1) Message routing/distribution/filter tool We need to copy messages from a set of input topics to a set of output topics based on their message key values. Each message in an input topic will go to 0 to N output topics, each output topic will receive messages from 0 to N input topics. So basically the tool acts as a message routing component in our system. Example configuration: input topic A:output topic K:key value 1,key value 2,key value 3 input topic A:output topic L:key value 2,key value 4 input topic B:output topic K:key value 5,key value 6 ... It would also be interesting to define distribution/filter rules based on regular expressions on the message key or message body. 2) Kafka-to-REST Push service We need to consume messages from a set of topics, translate them into REST web service calls and forward the data to existing, non-Kafka-aware systems with REST APIs that way. 3) HTTP REST API for consumers and producers We need to expose the simple consumer and the producer functionalities via REST web service calls, with authentication and per-topic-authorization on REST API level and TLS for transport encryption. Offset tracking is done by the connected systems, not by the broker/zookeeper/REST API. We expect a high message volume in the future here, so performance would be a key concern. Greetings Valentin -- http://khangaonkar.blogspot.com/ -- Thanks, Nageswara Rao.V *The LORD reigns*
Re: Message routing, Kafka-to-REST and HTTP API tools/frameworks for Kafka?
For 3, Confluent wrote a REST proxy that's pretty comprehensive. See the docs: http://confluent.io/docs/current/kafka-rest/docs/intro.html and a blog post describing it + future directions: http://blog.confluent.io/2015/03/25/a-comprehensive-open-source-rest-proxy-for-kafka/ There are a few other REST proxies: https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-HTTPREST But I don't think any of them support everything you need yet -- specifically, security stuff isn't included in any of them yet. You could address this with some performance hit by putting another HTTP server like nginx in front of the proxies (locally with each instance) to get that control. For Confluent's proxy, we're also thinking about how to add security features since we'll need something to protect admin operations: https://github.com/confluentinc/kafka-rest/wiki/Project---Admin-APIs -Ewen On Tue, Mar 24, 2015 at 11:34 PM, Nagesh nageswara.r...@gmail.com wrote: Hi, I think for 2) you can use Kafka Consumer and push messages to vertex event bus, which already have REST implementation (vertx-jersey). I would say, Vertx cluster can be used as receive data irrespective of topic and then publish to particular kafka topic. Then consume messages from kafka by different consumer and distribute. Kafka can hold messages without dropping at bursts and even at the time down stream slows down. Regards, Nageswara Rao On Wed, Mar 25, 2015 at 10:58 AM, Manoj Khangaonkar khangaon...@gmail.com wrote: Hi, For (1) and perhaps even for (2) where distribution/filtering on scale is required, I would look at using Apache Storm with kafka. For (3) , it seems you just need REST services wrapping kafka consumers/producers. I would start with usual suspects like jersey. regards On Tue, Mar 24, 2015 at 12:06 PM, Valentin kafka-...@sblk.de wrote: Hi guys, we have three Kafka use cases for which we have written our own PoC implementations, but where I am wondering whether there might be any fitting open source solution/tool/framework out there. Maybe someone of you has some ideas/pointers? :) 1) Message routing/distribution/filter tool We need to copy messages from a set of input topics to a set of output topics based on their message key values. Each message in an input topic will go to 0 to N output topics, each output topic will receive messages from 0 to N input topics. So basically the tool acts as a message routing component in our system. Example configuration: input topic A:output topic K:key value 1,key value 2,key value 3 input topic A:output topic L:key value 2,key value 4 input topic B:output topic K:key value 5,key value 6 ... It would also be interesting to define distribution/filter rules based on regular expressions on the message key or message body. 2) Kafka-to-REST Push service We need to consume messages from a set of topics, translate them into REST web service calls and forward the data to existing, non-Kafka-aware systems with REST APIs that way. 3) HTTP REST API for consumers and producers We need to expose the simple consumer and the producer functionalities via REST web service calls, with authentication and per-topic-authorization on REST API level and TLS for transport encryption. Offset tracking is done by the connected systems, not by the broker/zookeeper/REST API. We expect a high message volume in the future here, so performance would be a key concern. Greetings Valentin -- http://khangaonkar.blogspot.com/ -- Thanks, Nageswara Rao.V *The LORD reigns* -- Thanks, Ewen
Re: Message routing, Kafka-to-REST and HTTP API tools/frameworks for Kafka?
Hi, For (1) and perhaps even for (2) where distribution/filtering on scale is required, I would look at using Apache Storm with kafka. For (3) , it seems you just need REST services wrapping kafka consumers/producers. I would start with usual suspects like jersey. regards On Tue, Mar 24, 2015 at 12:06 PM, Valentin kafka-...@sblk.de wrote: Hi guys, we have three Kafka use cases for which we have written our own PoC implementations, but where I am wondering whether there might be any fitting open source solution/tool/framework out there. Maybe someone of you has some ideas/pointers? :) 1) Message routing/distribution/filter tool We need to copy messages from a set of input topics to a set of output topics based on their message key values. Each message in an input topic will go to 0 to N output topics, each output topic will receive messages from 0 to N input topics. So basically the tool acts as a message routing component in our system. Example configuration: input topic A:output topic K:key value 1,key value 2,key value 3 input topic A:output topic L:key value 2,key value 4 input topic B:output topic K:key value 5,key value 6 ... It would also be interesting to define distribution/filter rules based on regular expressions on the message key or message body. 2) Kafka-to-REST Push service We need to consume messages from a set of topics, translate them into REST web service calls and forward the data to existing, non-Kafka-aware systems with REST APIs that way. 3) HTTP REST API for consumers and producers We need to expose the simple consumer and the producer functionalities via REST web service calls, with authentication and per-topic-authorization on REST API level and TLS for transport encryption. Offset tracking is done by the connected systems, not by the broker/zookeeper/REST API. We expect a high message volume in the future here, so performance would be a key concern. Greetings Valentin -- http://khangaonkar.blogspot.com/
Message routing, Kafka-to-REST and HTTP API tools/frameworks for Kafka?
Hi guys, we have three Kafka use cases for which we have written our own PoC implementations, but where I am wondering whether there might be any fitting open source solution/tool/framework out there. Maybe someone of you has some ideas/pointers? :) 1) Message routing/distribution/filter tool We need to copy messages from a set of input topics to a set of output topics based on their message key values. Each message in an input topic will go to 0 to N output topics, each output topic will receive messages from 0 to N input topics. So basically the tool acts as a message routing component in our system. Example configuration: input topic A:output topic K:key value 1,key value 2,key value 3 input topic A:output topic L:key value 2,key value 4 input topic B:output topic K:key value 5,key value 6 ... It would also be interesting to define distribution/filter rules based on regular expressions on the message key or message body. 2) Kafka-to-REST Push service We need to consume messages from a set of topics, translate them into REST web service calls and forward the data to existing, non-Kafka-aware systems with REST APIs that way. 3) HTTP REST API for consumers and producers We need to expose the simple consumer and the producer functionalities via REST web service calls, with authentication and per-topic-authorization on REST API level and TLS for transport encryption. Offset tracking is done by the connected systems, not by the broker/zookeeper/REST API. We expect a high message volume in the future here, so performance would be a key concern. Greetings Valentin
