Do you have some idea of the size and number of messages per second you'll put
onto the topics at peak?
From: Sachin Mittal [mailto:sjmit...@gmail.com]
Sent: Thursday, December 1, 2016 9:44 AM
Subject: Re: I need some help with the production server architecture
And what about my brokers. Should I hedge them as well.
Like say put 2 zk on nodejs server and 1 on db server.
Put 2 brokers on db server and 1 on nodejs server, something like that.
On Thu, Dec 1, 2016 at 8:59 PM, Tauzell, Dave <dave.tauz...@surescripts.com>
> For low volume zookeeper doesn't seem to use many resources. I would put
> it on nodejs server as that will have less IO and heavy IO could
> impact zookeeper. Or, you could put some ZK nodes on nodejs and some on DB
> servers to hedge your bets. As always, you'll find out a lot once you
> actually start running it in production.
> -----Original Message-----
> From: Sachin Mittal [mailto:sjmit...@gmail.com]
> Sent: Thursday, December 1, 2016 6:03 AM
> To: firstname.lastname@example.org
> Subject: Re: I need some help with the production server architecture
> Folks any help on this.
> Just to put it in simple terms, since we have limited resources
> available to us what is better option 1. run zookeeper on servers
> running the nodejs web server or db server.
> 2. what about kafka brokers.
> On Tue, Nov 29, 2016 at 1:06 PM, Sachin Mittal <sjmit...@gmail.com> wrote:
> > Hi,
> > Sometime back i was informed on the group that in production we
> > should never run kafka on same physical machine. So based on that I
> > have a question on how to divide the server nodes we have to run
> > zookeper and kafka brokers.
> > I have a following setup
> > Data center 1
> > Lan 1 (3 VMs)
> > 192.168.xx.yy1
> > 192.168.xx.yy2
> > 192.168.xx.yy3
> > Right now here we are running a cluster of 3 nodejs web servers.
> > These collect data from web and write to kafka queue. Each VM has 70
> > GB of space.
> > Lan 2 (3 VMs)
> > 192.168.zz.aa1
> > 192.168.zz.aa2
> > 192.168.zz.aa3
> > These are served the cluster of our database server. Each VM has 400
> > GB of space.
> > Date center 2
> > Lan 1 (3 VMs)
> > 192.168.yy.bb1
> > 192.168.yy.bb2
> > 192.168.yy.bb3
> > Three new machines where we plan to run a cluster of new database to
> > be served as sink of kafka stream applications. Each VM has 400 GB
> > of
> > These have connectivity only between Lan 2 of Data center 1 with a
> > 100MBs of data transfer rate.
> > Each VM has a 4 core processor and 16 GB of RAM. They all run linux.
> > Now I would like my topics to be replicated with a factor of 3.
> > Since we don't foresee much volume of data, I don't want it to be
> > partitioned.
> > Also we would like one server to be used as streaming application
> > server, where we can run one or more kafka stream applications to
> > process the topics and write to the new database.
> > So please let me know what is a suitable division to run brokers
> > and zookeeper.
> > Thanks
> > Sachin
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of
> the individual or entity to whom they are addressed. If you have
> received this e-mail in error, please notify the sender by reply
> e-mail immediately and destroy all copies of the e-mail and any attachments.
This e-mail and any files transmitted with it are confidential, may contain
sensitive information, and are intended solely for the use of the individual or
entity to whom they are addressed. If you have received this e-mail in error,
please notify the sender by reply e-mail immediately and destroy all copies of
the e-mail and any attachments.