Apologies for the premature send. ---- Unfortunately, there is no API support for this right now. You could implement it yourself by implementing your own receiver and controlling the rate at which objects are "received". If you are using any of the standard receivers (Flume, Kafka, etc.), I recommended looking at the source code of the corresponding receiver and making your own version of Flume receiver / Kafka receiver.
Alternatively, there is a open JIRA<https://issues.apache.org/jira/browse/SPARK-1341> about this implementing this functionality. You could give it a shot at implementing this in a generic that it can be used for all receivers ;) TD On Wed, May 21, 2014 at 12:57 AM, Tathagata Das <tathagata.das1...@gmail.com > wrote: > Unfortunately, there is no API support for this right now. You could > implement it yourself by implementing your own receiver and controlling the > rate at which objects are "received". If you are using any of the standard > receivers (Flume, Kafka, etc.), I recommended looking at the source code of > the corresponding receiver and making your own version of > > Alternatively, there is a open > JIRA<https://issues.apache.org/jira/browse/SPARK-1341>about this implementing > this functionality. You could give it a shot at > implementing this in a generic that it can be used for all receivers ;) > > > > > On Tue, May 20, 2014 at 8:31 PM, Francis.Hu > <francis...@reachjunction.com>wrote: > >> sparkers, >> >> >> >> Is there a better way to control memory usage when streaming input's >> speed is faster than the speed of handled by spark streaming ? >> >> >> >> Thanks, >> >> Francis.Hu >> > >
