Hi guys, we have three Kafka use cases for which we have written our own PoC implementations, but where I am wondering whether there might be any fitting open source solution/tool/framework out there. Maybe someone of you has some ideas/pointers? :)
1) Message routing/distribution/filter tool We need to copy messages from a set of input topics to a set of output topics based on their message key values. Each message in an input topic will go to 0 to N output topics, each output topic will receive messages from 0 to N input topics. So basically the tool acts as a message routing component in our system. Example configuration: <input topic A>:<output topic K>:<key value 1>,<key value 2>,<key value 3> <input topic A>:<output topic L>:<key value 2>,<key value 4> <input topic B>:<output topic K>:<key value 5>,<key value 6> ... It would also be interesting to define distribution/filter rules based on regular expressions on the message key or message body. 2) Kafka-to-REST Push service We need to consume messages from a set of topics, translate them into REST web service calls and forward the data to existing, non-Kafka-aware systems with REST APIs that way. 3) HTTP REST API for consumers and producers We need to expose the simple consumer and the producer functionalities via REST web service calls, with authentication and per-topic-authorization on REST API level and TLS for transport encryption. Offset tracking is done by the connected systems, not by the broker/zookeeper/REST API. We expect a high message volume in the future here, so performance would be a key concern. Greetings Valentin
