The first level Kafkas are actually two separate clusters, and do about avg 50k messages/s, max 120k messages/s on 4 node clusters. The second level cluster doesn't handle all that load, as we filter down to the subset of our customers using the service. We currently use a retention window of 7 days/100GB and double replication. Most of the time end-to-end is less than 500ms, but that's subject to the latency of the decoration services.
On Fri, Oct 9, 2015 at 1:23 PM, Prabhjot Bharaj <prabhbha...@gmail.com> wrote: > Hi, > > Nice to know another usage of Kafka > > Could you tell how much load does the first and second level Kafka clusters > intake? > > What is your write throughput, and end-to-end latency, for how long do you > hold data in Kafka and how many minimum in-sync replicas you've chosen? > > Thanks, > Prabhjot > On Oct 10, 2015 1:41 AM, "Cory Kolbeck" <ckolb...@gmail.com> wrote: > > > Hi folks, > > > > I thought this list might find a case study interesting. Urban Airship > just > > released a new product, Connect, that uses Kafka heavily. We released > some > > discussion of the why and how we built it at > > https://www.urbanairship.com/blog/why-we-built-urban-airship-connect and > > https://www.urbanairship.com/blog/how-we-built-urban-airship-connect > > respectively. The posts aren't hugely in depth engineering wise, but I > hope > > folks find them interesting. > > > > Cory Kolbeck > > Senior Data Engineer > > Urban Airship > > >
