Dachuan, you may have correctly answered your own question. See Fig. 3 of the same paper, where "infinity" occurs in the vertical direction.
-- Christopher T. Nguyen Co-founder & CEO, Adatao <http://adatao.com> linkedin.com/in/ctnguyen On Sat, Nov 2, 2013 at 7:51 AM, dachuan <hdc1...@gmail.com> wrote: > Hi, developers, > > I found this sentence hard to understand, it's from sosp'13 spark streaming > paper: > > "Lineage cutoff: Because lineage graphs between RDDs > in D-Streams can grow indefinitely, we modified the > scheduler to forget lineage after an RDD has been checkpointed, > so that its state does not grow arbitrarily." > > In my personal understanding, the length of DStream chain is fixed, so the > RDDs these DStreams generate also have fixed length. Besides, the RDDs > don't depend on the RDDs in the previous round. So why does it claim that > lineage graph can grow indefinitely? when you say "grow indefinitely", do > you refer to lineage graph's width or the number of lineage graphs? > > thanks, > dachuan. >
