Re: Spark streaming and rate limit

Ok, I'll try to start from that when I'll try to implement it. Thanks again for the great support! Best, Flavio On Thu, Jun 19, 2014 at 10:57 AM, Michael Cutler wrote: > Hi Flavio, > > When your streaming job starts somewhere in the cluster the Receiver will > be started in its own thread/proc

Re: Spark streaming and rate limit

Hi Flavio, When your streaming job starts somewhere in the cluster the Receiver will be started in its own thread/process. You can do whatever you like within the receiver e.g. start and manage your own thread pool to fetch external data and feed Spark. If your Receiver dies Spark will re-provis

Re: Spark streaming and rate limit

Hi Michael, thanks for the tip, it's really an elegant solution. What I'm still missing here (maybe I should take a look at the code of TwitterInputDStream ..) i

Re: Spark streaming and rate limit

Hello Flavio, It sounds to me like the best solution for you is to implement your own ReceiverInputDStream/Receiver component to feed Spark Streaming with DStreams. It is not as scary as it sounds, take a look at some of the examples like TwitterInputDStream

Re: Spark streaming and rate limit

Yes, I need to call the external service for every event and the order does not matter. There's no time limit in which each events should be processed. I can't tell the producer to slow down nor drop events. Of course I could put a message broker in between like an AMQP or JMS broker but I was thin

Re: Spark streaming and rate limit

Flavio - i'm new to Spark as well but I've done stream processing using other frameworks. My comments below are not spark-streaming specific. Maybe someone who know more can provide better insights. I read your post on my phone and I believe my answer doesn't completely address the issue you have

Re: Spark streaming and rate limit

Thanks for the quick reply soumya. Unfortunately I'm a newbie with Spark..what do you mean? is there any reference to how to do that? On Thu, Jun 19, 2014 at 12:24 AM, Soumya Simanta wrote: > > You can add a back pressured enabled component in front that feeds data > into Spark. This component c

Re: Spark streaming and rate limit

You can add a back pressured enabled component in front that feeds data into Spark. This component can control in input rate to spark. > On Jun 18, 2014, at 6:13 PM, Flavio Pompermaier wrote: > > Hi to all, > in my use case I'd like to receive events and call an external service as > they pa

Spark streaming and rate limit

Hi to all, in my use case I'd like to receive events and call an external service as they pass through. Is it possible to limit the number of contemporaneous call to that service (to avoid DoS) using Spark streaming? if so, limiting the rate implies a possible buffer growth...how can I control the