If you want to separate your main application from producer logic you can use or write something similar to https://github.com/mozilla-metrics/bagheera . Basically have a service that provides an REST api which your main application can call and has a Kafkaproducer to write to kafka. -Harsha
On Tue, Nov 25, 2014, at 06:47 PM, Jun Rao wrote: > Assuming that you use a single producer in the async mode, the Kafka > overhead should be limited to a single thread. Using a cheaper > compression > codec such as snappy will also help reducing the CPU load. > > Thanks, > > Jun > > On Tue, Nov 25, 2014 at 12:02 PM, Krishna Raj > <reach.krishna...@gmail.com> > wrote: > > > Hi Jun, > > > > Thanks for replying back on this. Appreciated. > > > > I do understand that the Kafka Client just needs a protocol compatibility > > with the Application which is producing the messages. > > > > To clarity a bit more: > > > > I witnessed a scenario where a large scale website uses the Kafka Library > > in their Web Application. So in this case, the Kafka libraries are tied to > > the Application which are served by Web Servers. > > > > So, When there was an issue caused by Kafka related to CPU usage, the team > > wanted to do a patch. In this case, in order to do a patch, they had to > > create a new WAR package and deploy again in Web Server which is a > > significant effort. > > > > I totally understand that having a layer like Logging service in between > > Kafka and the Application will totally defect the purpose for Kafka. > > > > And I would love to know your advice how best to handle these type of > > maintenance. > > > > Thanks, > > Krishna Raj > > > > > > > > > > > > On Tue, Nov 25, 2014 at 10:58 AM, Jun Rao <jun...@gmail.com> wrote: > > > >> Could you be a bit more specific about the issue? As long as there is > >> protocol compatibility btw the Kafka client and the broker, upgrading the > >> Kafka client library should be easy, right? > >> > >> Thanks, > >> > >> Jun > >> > >> On Mon, Nov 24, 2014 at 3:57 PM, Krishna Raj <reach.krishna...@gmail.com> > >> wrote: > >> > >>> Hello Amazing Kafka Creators & User, > >>> > >>> I have learnt and use kafka in our Production system, so you can count > >>> my understanding as intermediate. > >>> > >>> With the statement that "Kafka has solved the Scalability and > >>> Availability needs for a large scale message publish/subscribe system", I > >>> understand that having a Producer Service which sits in between the > >>> Application and the Producer defects the one major purpose of Kafka. > >>> > >>> So, my question is, How to loosely couple Kafka with my Production > >>> Application ? > >>> > >>> The reason being, I wish to do all producer code and Kafka library > >>> maintenance without affecting my large scale Production system. Its not an > >>> easy thing to buy a window to these type of changes done on a large scale > >>> production application :) > >>> > >>> Any advice on how this can be achieved(even moderately) will greatly > >>> help ? > >>> > >>> Thanks, > >>> Krishna Raj > >>> > >> > >> > >
