Gourav, Yes, sorry. Apparently I failed to mention I'm having these problems with Spark consuming from a kinesis stream. Been putting in late nights to figure this out and it's affecting my brain. :^)
-jeremy -- Jeremy Kelley | Technical Director, Data jkel...@carbonblack.com | Carbon Black Threat Engineering > On Dec 15, 2017, at 9:12 AM, Gourav Sengupta <gourav.sengu...@gmail.com> > wrote: > > Hi Jeremy, > > just out of curiosity - you do know that this is a SPARK user group? > > > Regards, > Gourav > > On Thu, Dec 14, 2017 at 7:03 PM, Jeremy Kelley <jkel...@carbonblack.com > <mailto:jkel...@carbonblack.com>> wrote: > We have a largeish kinesis stream with about 25k events per second and each > record is around 142k. I have tried multiple cluster sizes, multiple batch > sizes, multiple parameters... I am doing minimal transformations on the > data. Whatever happens I can sustain consuming 25k with minimal effort and > cluster load for about 5-10 minutes and then always always the stream shapes > down and hovers around 5k EPS. > > I can give MANY more details but I was curious if anyone had seen similar > behavior. > > Thanks, > Jeremy > > > -- > Jeremy Kelley | Technical Director, Data > jkel...@carbonblack.com <mailto:jkel...@carbonblack.com> | Carbon Black > Threat Engineering > > >
smime.p7s
Description: S/MIME cryptographic signature
