Should I use many kafka brokers or one will sufice? Thanks
On Wed, Aug 13, 2014 at 7:24 AM, Telles Nobrega <[email protected]> wrote: > It could be just one job, but what is the best way to deploy many > instances of this job so I could process a heavy load of messages? > > Thanks, > > On 13 Aug 2014, at 01:39, Yan Fang <[email protected]> wrote: > > > *"Does one kafka-broker handle this much messages per second?"* > > > > I believe @Chris has better answer about this. > > > > > > > > *"I have one job that get this messages and another that reads from the > > output of the first job that does some more processing."* > > > > Why not use one job get messages and process them? > > > > *" when I change a* > > > > *configuration of one my jobs do I need to recompile it and send the new > > tar.gz to hdfs or just change the deploy/samza config and it should > work."* > > > > No, you don't need to recompile. Change the config and run-job. It > will > > work. > > > > Thanks. > > > > Cheers, > > > > Fang, Yan > > [email protected] > > +1 (206) 849-4108 > > > > > > On Tue, Aug 12, 2014 at 8:47 PM, Telles Nobrega <[email protected] > > > > wrote: > > > >> Not completely related to the topic of the question but when I change a > >> configuration of one my jobs do I need to recompile it and send the new > >> tar.gz to hdfs or just change the deploy/samza config and it should > work. > >> > >> Thanks > >> > >> > >> On Tue, Aug 12, 2014 at 11:23 PM, Telles Nobrega < > [email protected]> > >> wrote: > >> > >>> Hi, I'm running an experiment that I'm suppose to run samza with > >> different > >>> input rates. First I'm running with 420 messages/second and I scale up > to > >>> 33200 messages/second. > >>> > >>> Does one kafka-broker handle this much messages per second? > >>> Second, what is the best way to read into samza this much messages? I > >> have > >>> one job that get this messages and another that reads from the output > of > >>> the first job that does some more processing. Is the best way to use > more > >>> containers and split kafka topics in partitions (the same number of > >>> containers) or is there a better way to do this. > >>> > >>> Thanks in advance, > >>> > >>> -- > >>> ------------------------------------------ > >>> Telles Mota Vidal Nobrega > >>> M.sc. Candidate at UFCG > >>> B.sc. in Computer Science at UFCG > >>> Software Engineer at OpenStack Project - HP/LSD-UFCG > >>> > >> > >> > >> > >> -- > >> ------------------------------------------ > >> Telles Mota Vidal Nobrega > >> M.sc. Candidate at UFCG > >> B.sc. in Computer Science at UFCG > >> Software Engineer at OpenStack Project - HP/LSD-UFCG > >> > > -- ------------------------------------------ Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG
