Tim, If the state is stored to local disk, other parties need to be on same node which may not be the case. If state is in hdfs it can be accessed from all the nodes in a cluster.
Thanks - Gaurav > On Nov 25, 2015, at 11:33 PM, Timothy Farkas <[email protected]> wrote: > > Chetan your use case is not valid, if checkpointed is called after the > operator state is stored to local disk a similar storage function could be > performed. It is not necessary to wait for that same state to be > asynchronously moved to hdfs. Please provide an example of a valid use case > :) > > +1 for fixing this bug/regression as Thomas suggested. > > On Wed, Nov 25, 2015 at 5:35 PM, Chandni Singh <[email protected]> > wrote: > >> Semver was broken with Async checkpointing. The behavior was changed as >> pointed out before in the discussion. Also making it difficult for operator >> developer doesn't give us anything. >> >> +1 for fixing it in the way Thomas suggested. >> >> On Wed, Nov 25, 2015 at 4:07 PM, Chetan Narsude (cnarsude) < >> [email protected]> wrote: >> >>> Yes - a few but cannot share the details - protected under NDA - ping me >>> in private and I can probably be able to give you more generic details on >>> similar cooked up examples. >>> >>> The part that follows “e.g.” below is an example that probably is >>> sufficient to infer the use case logically, I think. I shared that to >>> exemplify how changing the semantics will break semver. >>> >>> — >>> Chetan >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On 11/25/15, 3:51 PM, "Thomas Weise" <[email protected]> wrote: >>> >>>> Do you have a specific example? >>>> >>>> I see this happening in committed(), but not in checkpointed() where the >>>> checkpoint remains intermediate, whether it was copied to HDFS or not. >>>> >>>> >>>> On Wed, Nov 25, 2015 at 3:42 PM, Chetan Narsude (cnarsude) < >>>> [email protected]> wrote: >>>> >>>>>> >>>>>> Until we have this, how about we restore the previous behavior >>>>>> temporarily? >>>>>> Calling checkpointed() immediately does not seem to pose any >> practical >>>>>> issue but ensures that the code that was written under this >> assumption >>>>> is >>>>>> not broken. >>>>> >>>>> We can¹t do it. It would be incorrect. It breaks all the other code >> that >>>>> (unassumingly) correctly complied to the semantics. e.g. an operator >>>>> which >>>>> informs interesting parties that the checkpointed data is available >> for >>>>> immediate consumption from storage. >>>>> >>>>> ‹ >>>>> Chetan >>>>> >>>>> >>> >>> >>
