Re: Urban Airship Connect

[Prabhjot Bharaj]

Hi,

Great info.
What is your approximate/average message size ?
And, how many topics and partitions do you use  in your 4-node cluster.
Also, please share your hardware specs

Thanks,
Prabhjot

On Sat, Oct 10, 2015 at 2:00 AM, Cory Kolbeck  wrote:

> The first level Kafkas are actually two separate clusters, and do about avg
> 50k messages/s, max 120k messages/s  on 4 node clusters. The second level
> cluster doesn't handle all that load, as we filter down to the subset of
> our customers using the service. We currently use a retention window of 7
> days/100GB and double replication. Most of the time end-to-end is less than
> 500ms, but that's subject to the latency of the decoration services.
>
> On Fri, Oct 9, 2015 at 1:23 PM, Prabhjot Bharaj 
> wrote:
>
> > Hi,
> >
> > Nice to know another usage of Kafka
> >
> > Could you tell how much load does the first and second level Kafka
> clusters
> > intake?
> >
> > What is your write throughput, and end-to-end latency, for how long do
> you
> > hold data in Kafka and how many minimum in-sync replicas you've chosen?
> >
> > Thanks,
> > Prabhjot
> > On Oct 10, 2015 1:41 AM, "Cory Kolbeck"  wrote:
> >
> > > Hi folks,
> > >
> > > I thought this list might find a case study interesting. Urban Airship
> > just
> > > released a new product, Connect, that uses Kafka heavily. We released
> > some
> > > discussion of the why and how we built it at
> > > https://www.urbanairship.com/blog/why-we-built-urban-airship-connect
> and
> > > https://www.urbanairship.com/blog/how-we-built-urban-airship-connect
> > > respectively. The posts aren't hugely in depth engineering wise, but I
> > hope
> > > folks find them interesting.
> > >
> > > Cory Kolbeck
> > > Senior Data Engineer
> > > Urban Airship
> > >
> >
>



-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

Re: Urban Airship Connect

[Prabhjot Bharaj]

Hi,

Nice to know another usage of Kafka

Could you tell how much load does the first and second level Kafka clusters
intake?

What is your write throughput, and end-to-end latency, for how long do you
hold data in Kafka and how many minimum in-sync replicas you've chosen?

Thanks,
Prabhjot
On Oct 10, 2015 1:41 AM, "Cory Kolbeck"  wrote:

> Hi folks,
>
> I thought this list might find a case study interesting. Urban Airship just
> released a new product, Connect, that uses Kafka heavily. We released some
> discussion of the why and how we built it at
> https://www.urbanairship.com/blog/why-we-built-urban-airship-connect and
> https://www.urbanairship.com/blog/how-we-built-urban-airship-connect
> respectively. The posts aren't hugely in depth engineering wise, but I hope
> folks find them interesting.
>
> Cory Kolbeck
> Senior Data Engineer
> Urban Airship
>

Re: Urban Airship Connect

[Cory Kolbeck]

The first level Kafkas are actually two separate clusters, and do about avg
50k messages/s, max 120k messages/s  on 4 node clusters. The second level
cluster doesn't handle all that load, as we filter down to the subset of
our customers using the service. We currently use a retention window of 7
days/100GB and double replication. Most of the time end-to-end is less than
500ms, but that's subject to the latency of the decoration services.

On Fri, Oct 9, 2015 at 1:23 PM, Prabhjot Bharaj 
wrote:

> Hi,
>
> Nice to know another usage of Kafka
>
> Could you tell how much load does the first and second level Kafka clusters
> intake?
>
> What is your write throughput, and end-to-end latency, for how long do you
> hold data in Kafka and how many minimum in-sync replicas you've chosen?
>
> Thanks,
> Prabhjot
> On Oct 10, 2015 1:41 AM, "Cory Kolbeck"  wrote:
>
> > Hi folks,
> >
> > I thought this list might find a case study interesting. Urban Airship
> just
> > released a new product, Connect, that uses Kafka heavily. We released
> some
> > discussion of the why and how we built it at
> > https://www.urbanairship.com/blog/why-we-built-urban-airship-connect and
> > https://www.urbanairship.com/blog/how-we-built-urban-airship-connect
> > respectively. The posts aren't hugely in depth engineering wise, but I
> hope
> > folks find them interesting.
> >
> > Cory Kolbeck
> > Senior Data Engineer
> > Urban Airship
> >
>