Another approach for Chetan's use case can be to extend AsyncFSStorageAgent and perform the function after copy to hdfs is completed. Benefit is that the function will be performed asynchronously (with copy to hdfs) and will not block operator's thread.
Chandni On Wed, Nov 25, 2015 at 11:33 PM, Timothy Farkas <[email protected]> wrote: > Chetan your use case is not valid, if checkpointed is called after the > operator state is stored to local disk a similar storage function could be > performed. It is not necessary to wait for that same state to be > asynchronously moved to hdfs. Please provide an example of a valid use case > :) > > +1 for fixing this bug/regression as Thomas suggested. > > On Wed, Nov 25, 2015 at 5:35 PM, Chandni Singh <[email protected]> > wrote: > > > Semver was broken with Async checkpointing. The behavior was changed as > > pointed out before in the discussion. Also making it difficult for > operator > > developer doesn't give us anything. > > > > +1 for fixing it in the way Thomas suggested. > > > > On Wed, Nov 25, 2015 at 4:07 PM, Chetan Narsude (cnarsude) < > > [email protected]> wrote: > > > > > Yes - a few but cannot share the details - protected under NDA - ping > me > > > in private and I can probably be able to give you more generic details > on > > > similar cooked up examples. > > > > > > The part that follows “e.g.” below is an example that probably is > > > sufficient to infer the use case logically, I think. I shared that to > > > exemplify how changing the semantics will break semver. > > > > > > — > > > Chetan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/25/15, 3:51 PM, "Thomas Weise" <[email protected]> wrote: > > > > > > >Do you have a specific example? > > > > > > > >I see this happening in committed(), but not in checkpointed() where > the > > > >checkpoint remains intermediate, whether it was copied to HDFS or not. > > > > > > > > > > > >On Wed, Nov 25, 2015 at 3:42 PM, Chetan Narsude (cnarsude) < > > > >[email protected]> wrote: > > > > > > > >> > > > > >> >Until we have this, how about we restore the previous behavior > > > >> >temporarily? > > > >> >Calling checkpointed() immediately does not seem to pose any > > practical > > > >> >issue but ensures that the code that was written under this > > assumption > > > >>is > > > >> >not broken. > > > >> > > > >> We can¹t do it. It would be incorrect. It breaks all the other code > > that > > > >> (unassumingly) correctly complied to the semantics. e.g. an operator > > > >>which > > > >> informs interesting parties that the checkpointed data is available > > for > > > >> immediate consumption from storage. > > > >> > > > >> ‹ > > > >> Chetan > > > >> > > > >> > > > > > > > > >
