Hi Prabhjot, Confluent has a REST proxy with docs that may give some guidance: http://docs.confluent.io/1.0/kafka-rest/docs/intro.html The new producer that it uses is very efficient, so you should be able to get pretty good throughput. You take a bit of a hit due to the overhead of sending data through a proxy, but with appropriate batching you can get about 2/3 the performance as you would get using the Java producer directly.
There are also a few other proxies you can find here: https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-HTTPREST You can also put nginx (or HAProxy, or a variety of other solutions) in front of REST proxies for load balancing, HA, SSL termination, etc. This is yet another hop, so it might affect throughput and latency. -Ewen On Mon, Jul 27, 2015 at 6:55 AM, Prabhjot Bharaj <prabhbha...@gmail.com> wrote: > Hi Folks, > > I would like to understand the best practices when using kafka as the > source-of-truth, given the fact that I want to pump in data to Kafka using > http methods. > > What are the current production configurations for such a use case:- > > 1. Kafka-http-client - is it scalable the way Nginx is ?? > 2. Using Kafka and Nginx together - If anybody has used this, please > explain > 3. Any other scalable method ? > > Regards, > prabcs > -- Thanks, Ewen
