Even if you don't need the checkpointing data for recovery, "mapWithState" still needs to use "checkpoint" to cut off the RDD lineage.
On Mon, Jan 23, 2017 at 12:30 AM, shyla deshpande <deshpandesh...@gmail.com> wrote: > Hello spark users, > > I do have the same question as Daniel. > > I would like to save the state in Cassandra and on failure recover using > the initialState. If some one has already tried this, please share your > experience and sample code. > > Thanks. > > On Thu, Nov 17, 2016 at 9:45 AM, Daniel Haviv < > daniel.ha...@veracity-group.com> wrote: > >> Hi, >> Is it possible to use mapWithState without checkpointing at all ? >> I'd rather have the whole application fail, restart and reload an >> initialState RDD than pay for checkpointing every 10 batches. >> >> Thank you, >> Daniel >> > >
