Hi, I think for 2) you can use Kafka Consumer and push messages to vertex event bus, which already have REST implementation (vertx-jersey).
I would say, Vertx cluster can be used as receive data irrespective of topic and then publish to particular kafka topic. Then consume messages from kafka by different consumer and distribute. Kafka can hold messages without dropping at bursts and even at the time down stream slows down. Regards, Nageswara Rao On Wed, Mar 25, 2015 at 10:58 AM, Manoj Khangaonkar <[email protected]> wrote: > Hi, > > For (1) and perhaps even for (2) where distribution/filtering on scale is > required, I would look at using Apache Storm with kafka. > > For (3) , it seems you just need REST services wrapping kafka > consumers/producers. I would start with usual suspects like jersey. > > regards > > On Tue, Mar 24, 2015 at 12:06 PM, Valentin <[email protected]> wrote: > > > > > Hi guys, > > > > we have three Kafka use cases for which we have written our own PoC > > implementations, > > but where I am wondering whether there might be any fitting open source > > solution/tool/framework out there. > > Maybe someone of you has some ideas/pointers? :) > > > > 1) Message routing/distribution/filter tool > > We need to copy messages from a set of input topics to a set of output > > topics > > based on their message key values. Each message in an input topic will go > > to 0 to N output topics, > > each output topic will receive messages from 0 to N input topics. > > So basically the tool acts as a message routing component in our system. > > Example configuration: > > <input topic A>:<output topic K>:<key value 1>,<key value 2>,<key value > 3> > > <input topic A>:<output topic L>:<key value 2>,<key value 4> > > <input topic B>:<output topic K>:<key value 5>,<key value 6> > > ... > > It would also be interesting to define distribution/filter rules based on > > regular expressions on the message key or message body. > > > > 2) Kafka-to-REST Push service > > We need to consume messages from a set of topics, translate them into > REST > > web service calls > > and forward the data to existing, non-Kafka-aware systems with REST APIs > > that way. > > > > 3) HTTP REST API for consumers and producers > > We need to expose the simple consumer and the producer functionalities > via > > REST web service calls, > > with authentication and per-topic-authorization on REST API level and TLS > > for transport encryption. > > Offset tracking is done by the connected systems, not by the > > broker/zookeeper/REST API. > > We expect a high message volume in the future here, so performance would > > be a key concern. > > > > Greetings > > Valentin > > > > > > -- > http://khangaonkar.blogspot.com/ > -- Thanks, Nageswara Rao.V *"The LORD reigns"*
