Hi Jeremy, just out of curiosity - you do know that this is a SPARK user group?
Regards, Gourav On Thu, Dec 14, 2017 at 7:03 PM, Jeremy Kelley <jkel...@carbonblack.com> wrote: > We have a largeish kinesis stream with about 25k events per second and > each record is around 142k. I have tried multiple cluster sizes, multiple > batch sizes, multiple parameters... I am doing minimal transformations on > the data. Whatever happens I can sustain consuming 25k with minimal effort > and cluster load for about 5-10 minutes and then always always the stream > shapes down and hovers around 5k EPS. > > I can give MANY more details but I was curious if anyone had seen similar > behavior. > > Thanks, > Jeremy > > > -- > Jeremy Kelley | Technical Director, Data > jkel...@carbonblack.com | Carbon Black Threat Engineering > > >
