OK - thanks a lot On Tue, Feb 24, 2015 at 9:49 AM, Tathagata Das <[email protected]> wrote:
> Yes. > > On Mon, Feb 23, 2015 at 11:16 PM, Avi Levi <[email protected]> wrote: > >> @Tathagata Das so basically you are saying it is supported out of the >> box, but we should expect a significant performance hit - is that right? >> >> >> >> On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das <[email protected]> >> wrote: >> >>> The default persistence level is MEMORY_AND_DISK, so the LRU policy >>> would discard the blocks to disk, so the streaming app will not fail. >>> However, since things will get constantly read in and out of disk as >>> windows are processed, the performance wont be great. So it is best to have >>> sufficient memory to keep all the window data in memory. >>> >>> TD >>> >>> On Mon, Feb 23, 2015 at 8:26 AM, Shao, Saisai <[email protected]> >>> wrote: >>> >>>> I don't think current Spark Streaming supports window operations which >>>> beyond its available memory, internally Spark Streaming puts all the data >>>> in the memory belongs to the effective window, if the memory is not enough, >>>> BlockManager will discard the blocks at LRU policy, so something unexpected >>>> will be occurred. >>>> >>>> Thanks >>>> Jerry >>>> >>>> -----Original Message----- >>>> From: avilevi3 [mailto:[email protected]] >>>> Sent: Monday, February 23, 2015 12:57 AM >>>> To: [email protected] >>>> Subject: spark streaming window operations on a large window size >>>> >>>> Hi guys, >>>> >>>> does spark streaming supports window operations on a sliding window >>>> that is data is larger than the available memory? >>>> we would like to >>>> currently we are using kafka as input, but we could change that if >>>> needed. >>>> >>>> thanks >>>> Avi >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] For >>>> additional commands, e-mail: [email protected] >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] >>>> For additional commands, e-mail: [email protected] >>>> >>>> >>> >> >
