You can throttle the no receiver direct Kafka stream using spark.streaming.kafka.maxRatePerPartition <http://spark.apache.org/docs/latest/configuration.html#spark-streaming>
On Wed, May 27, 2015 at 4:34 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Have you seen > http://stackoverflow.com/questions/29051579/pausing-throttling-spark-spark-streaming-application > ? > > Cheers > > On Wed, May 27, 2015 at 4:11 PM, dgoldenberg <dgoldenberg...@gmail.com> > wrote: > >> Hi, >> >> With the no receivers approach to streaming from Kafka, is there a way to >> set something like spark.streaming.receiver.maxRate so as not to overwhelm >> the Spark consumers? >> >> What would be some of the ways to throttle the streamed messages so that >> the >> consumers don't run out of memory? >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-from-Kafka-no-receivers-and-spark-streaming-receiver-maxRate-tp23061.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
