The default persistence level is MEMORY_AND_DISK, so the LRU policy would discard the blocks to disk, so the streaming app will not fail. However, since things will get constantly read in and out of disk as windows are processed, the performance wont be great. So it is best to have sufficient memory to keep all the window data in memory.
TD On Mon, Feb 23, 2015 at 8:26 AM, Shao, Saisai <saisai.s...@intel.com> wrote: > I don't think current Spark Streaming supports window operations which > beyond its available memory, internally Spark Streaming puts all the data > in the memory belongs to the effective window, if the memory is not enough, > BlockManager will discard the blocks at LRU policy, so something unexpected > will be occurred. > > Thanks > Jerry > > -----Original Message----- > From: avilevi3 [mailto:avile...@gmail.com] > Sent: Monday, February 23, 2015 12:57 AM > To: user@spark.apache.org > Subject: spark streaming window operations on a large window size > > Hi guys, > > does spark streaming supports window operations on a sliding window that > is data is larger than the available memory? > we would like to > currently we are using kafka as input, but we could change that if needed. > > thanks > Avi > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional > commands, e-mail: user-h...@spark.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
