Hi, Great info. What is your approximate/average message size ? And, how many topics and partitions do you use in your 4-node cluster. Also, please share your hardware specs
Thanks, Prabhjot On Sat, Oct 10, 2015 at 2:00 AM, Cory Kolbeck <ckolb...@gmail.com> wrote: > The first level Kafkas are actually two separate clusters, and do about avg > 50k messages/s, max 120k messages/s on 4 node clusters. The second level > cluster doesn't handle all that load, as we filter down to the subset of > our customers using the service. We currently use a retention window of 7 > days/100GB and double replication. Most of the time end-to-end is less than > 500ms, but that's subject to the latency of the decoration services. > > On Fri, Oct 9, 2015 at 1:23 PM, Prabhjot Bharaj <prabhbha...@gmail.com> > wrote: > > > Hi, > > > > Nice to know another usage of Kafka > > > > Could you tell how much load does the first and second level Kafka > clusters > > intake? > > > > What is your write throughput, and end-to-end latency, for how long do > you > > hold data in Kafka and how many minimum in-sync replicas you've chosen? > > > > Thanks, > > Prabhjot > > On Oct 10, 2015 1:41 AM, "Cory Kolbeck" <ckolb...@gmail.com> wrote: > > > > > Hi folks, > > > > > > I thought this list might find a case study interesting. Urban Airship > > just > > > released a new product, Connect, that uses Kafka heavily. We released > > some > > > discussion of the why and how we built it at > > > https://www.urbanairship.com/blog/why-we-built-urban-airship-connect > and > > > https://www.urbanairship.com/blog/how-we-built-urban-airship-connect > > > respectively. The posts aren't hugely in depth engineering wise, but I > > hope > > > folks find them interesting. > > > > > > Cory Kolbeck > > > Senior Data Engineer > > > Urban Airship > > > > > > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"
