Gaurav, Chandni's method would address your point. Or you can copy the state wherever you want (even asynchronously) from the checkpointed callback.
On Wed, Nov 25, 2015 at 11:47 PM, Chandni Singh <[email protected]> wrote: > Another approach for Chetan's use case can be to extend AsyncFSStorageAgent > and perform the function after copy to hdfs is completed. > Benefit is that the function will be performed asynchronously (with copy to > hdfs) and will not block operator's thread. > > Chandni > > On Wed, Nov 25, 2015 at 11:33 PM, Timothy Farkas <[email protected]> > wrote: > > > Chetan your use case is not valid, if checkpointed is called after the > > operator state is stored to local disk a similar storage function could > be > > performed. It is not necessary to wait for that same state to be > > asynchronously moved to hdfs. Please provide an example of a valid use > case > > :) > > > > +1 for fixing this bug/regression as Thomas suggested. > > > > On Wed, Nov 25, 2015 at 5:35 PM, Chandni Singh <[email protected]> > > wrote: > > > > > Semver was broken with Async checkpointing. The behavior was changed as > > > pointed out before in the discussion. Also making it difficult for > > operator > > > developer doesn't give us anything. > > > > > > +1 for fixing it in the way Thomas suggested. > > > > > > On Wed, Nov 25, 2015 at 4:07 PM, Chetan Narsude (cnarsude) < > > > [email protected]> wrote: > > > > > > > Yes - a few but cannot share the details - protected under NDA - ping > > me > > > > in private and I can probably be able to give you more generic > details > > on > > > > similar cooked up examples. > > > > > > > > The part that follows “e.g.” below is an example that probably is > > > > sufficient to infer the use case logically, I think. I shared that to > > > > exemplify how changing the semantics will break semver. > > > > > > > > — > > > > Chetan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/25/15, 3:51 PM, "Thomas Weise" <[email protected]> wrote: > > > > > > > > >Do you have a specific example? > > > > > > > > > >I see this happening in committed(), but not in checkpointed() where > > the > > > > >checkpoint remains intermediate, whether it was copied to HDFS or > not. > > > > > > > > > > > > > > >On Wed, Nov 25, 2015 at 3:42 PM, Chetan Narsude (cnarsude) < > > > > >[email protected]> wrote: > > > > > > > > > >> > > > > > >> >Until we have this, how about we restore the previous behavior > > > > >> >temporarily? > > > > >> >Calling checkpointed() immediately does not seem to pose any > > > practical > > > > >> >issue but ensures that the code that was written under this > > > assumption > > > > >>is > > > > >> >not broken. > > > > >> > > > > >> We can¹t do it. It would be incorrect. It breaks all the other > code > > > that > > > > >> (unassumingly) correctly complied to the semantics. e.g. an > operator > > > > >>which > > > > >> informs interesting parties that the checkpointed data is > available > > > for > > > > >> immediate consumption from storage. > > > > >> > > > > >> ‹ > > > > >> Chetan > > > > >> > > > > >> > > > > > > > > > > > > > >
