I had a workaround for exactly the same scenario http://apache-spark-developers-list.1001551.n3.nabble.com/SparkStreaming-Workaround-for-BlockNotFound-Exceptions-td12096.html
Apart from that, if you are using this consumer https://github.com/dibbhatt/kafka-spark-consumer it also has a built-in rate limiting, Also in Spark 1.5.0 they have a rate limiting/back-pressure (haven't tested it on production though). Thanks Best Regards On Tue, Sep 15, 2015 at 11:56 PM, Michal Čizmazia <mici...@gmail.com> wrote: > Hi, > > I have a Reliable Custom Receiver storing messages into Spark. Is there > way how to prevent my receiver from storing more messages into Spark when > the Scheduling Delay reaches a certain threshold? > > Possible approaches: > #1 Does Spark block on the Receiver.store(messages) call to prevent > storing more messages and overflowing the system? > #2 How to obtain the Scheduling Delay in the Custom Receiver, so that I > can implement the feature. > > Thanks, > > Mike > >
