We're using AWS ECS for our Kafka cluster of six nodes. We did some performance testing on a three node cluster and the results were as good as the Linkedin published results on bare metal machines.
We are using EBS st1 drives. The bottleneck is the network to the ebs volumes. So for about 25% more cost, we doubled our vms, using twice as many 1/2 sized EBS volumes. -Christian On Fri, Jul 8, 2016 at 12:07 PM Krish <krishnan.k.i...@gmail.com> wrote: > Thanks, Christian. > I am currently reading about kafka-on-mesos. > I will hack something this weekend to see if I can bring up a kafka > scheduler on mesos using dockerized brokers. <fingers crossed>. > > > > -- > κρισhναν > > On Thu, Jul 7, 2016 at 7:29 PM, Christian Posta <christian.po...@gmail.com > > > wrote: > > > One thing I can think of is Kafka likes lots of OS page cache. > Dockerizing > > from the standpoint of packaging configs is a good idea, just make sure > if > > you're running many brokers together on the same host, they've got enough > > resources (CPU/Mem) so they don't starve each other. > > > > On Thu, Jul 7, 2016 at 2:30 AM, Krish <krishnan.k.i...@gmail.com> wrote: > > > >> Hi, > >> I am currently testing a custom docker volume driver plugin for AWS > >> EFS/EBS > >> access and mounting. So, running kafka broker inside a container makes > >> will > >> ease up a lot of configuration issues wrt storage for me. > >> > >> Are there any pros and cons of dockerizing kafka broker? > >> Off the top of my head, since kafka forms the base of our setup, I can > >> think of making is use the host networking stack, and increase ulimits > for > >> the container. > >> I would like to know if and when kafka becomes greedy and cannibalizes > >> resources; I can also ensure that it runs on a dedicated machine. > >> > >> Thanks. > >> > >> Best, > >> Krish > >> > > > > > > > > -- > > *Christian Posta* > > twitter: @christianposta > > http://www.christianposta.com/blog > > http://fabric8.io > > > > >
