Re:Re: How big the spark stream window could be ?

Thanks. What if I use batch calculation instead of stream computing? Do I still need that much memory? For example, if the 24 hour data set is 100 GB. Do I also need a 100GB RAM to do the one time batch calculation ? At 2016-05-09 15:14:47, "Saisai Shao" wrote:

Re:Re: How big the spark stream window could be ?

Thanks Mich I guess I did not make my question clear enough. I know the terms like interval or window. I also know how to use them. The problem is that in my case, I need to set the window to cover data for 24 hours or 1 hours. I am not sure if it is a good way because the window is just too