I'm interested in understanding this as well. One of the main ways Tachyon is supposed to realize performance gains without sacrificing durability is by storing the lineage of data rather than full copies of it (similar to Spark). But if Spark isn't sending lineage information into Tachyon, then I'm not sure how this isn't a durability concern.
On Wed, Dec 10, 2014 at 5:47 AM, Jun Feng Liu <liuj...@cn.ibm.com> wrote: > Dose Spark today really leverage Tachyon linage to process data? It seems > like the application should call createDependency function in TachyonFS > to create a new linage node. But I did not find any place call that in > Spark code. Did I missed anything? > > Best Regards > > > *Jun Feng Liu* > IBM China Systems & Technology Laboratory in Beijing > > ------------------------------ > [image: 2D barcode - encoded with contact information] *Phone: > *86-10-82452683 > > * E-mail:* *liuj...@cn.ibm.com* <liuj...@cn.ibm.com> > [image: IBM] > > BLD 28,ZGC Software Park > No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 > China > > > > >
