Hi Fernando,
the "persistence" of a DStream is defined depending of the StorageLevel.
Window is not related to persistence: it's the processing of multiple
DStream in one, a kind of "gather of DStreams". The transformation is
applied on a "slide window". For instance, you define a window of 3
DStreams, and you apply the transformation on this window.
I hope it helps.
Regards
JB
On 11/16/2015 07:38 PM, Fernando O. wrote:
Hi all,
I was wondering if someone could give me a brief explanation or
point me in the right direction in the code for where DStream
persistence is donde.
I'm looking at DStream.java but all it does is setting the StorageLevel,
and neither WindowedDStream or ReducedWindowedDStream seem to change
that behaviour a lot:
WindowedDStream delegates on the parent and
ReducedWindowedDStream calls super and delegates on an aggregated DStream
So it seems like persist will only change the StorageLevel and then you
need to compute the DStream to persist the inner RDDs
So basically what I'm trying to verify is: persist is just lazy
persistence method that sets the Storage level and the actual
persistence takes place when you compute the DStream right?
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
