There is a problem here, keeping old checkpoints and recovering from them means preserving the old input data along with the state. This is more than the mechanism of actually creating named checkpoints, it means having the ability for operators to move forward (a.k.a committed and dropping committed states and buffer data) while still having the ability to replay from that point from the input source and providing a way for operators (at first look input operators) to distinguish that. Why would someone need this with idempotent processing? Is there a specific use case you are looking at? Suppose we go do this, for the mechanism, I would be in favor of reusing existing tuple.
On Thu, Aug 4, 2016 at 8:44 AM, Vlad Rozov <v.ro...@datatorrent.com> wrote: > +1 for the feature. At first look I am more in favor of reusing existing > control tuple. > > Thank you, > > Vlad > > > On 8/4/16 08:17, Sandesh Hegde wrote: > >> @Chinmay >> We can enhance the existing checkpoint tuple but that one is more >> frequently used than this feature, so why burden Checkpoint tuple with >> an extra field? >> >> @Aniruddha >> It is better to leave the scheduling to the users, they can use any tool >> that they are already familiar with. >> >> On Thu, Aug 4, 2016 at 7:40 AM Aniruddha Thombare < >> anirud...@datatorrent.com> >> wrote: >> >> +1 On the idea, it would be awesome to have. >>> >>> Question: Can we further develop this brilliant idea into:- >>> Scheduled checkpoints ( To save as dynamically named checkpoint)? >>> This would be on the lines of logrotate / general backup strategies. >>> >>> >>> Thanks, >>> >>> A >>> >>> _____________________________________ >>> Sent with difficulty, I mean handheld ;) >>> On 4 Aug 2016 8:03 pm, "Munagala Ramanath" <r...@datatorrent.com> wrote: >>> >>> +1 >>>> >>>> Ram >>>> >>>> On Thu, Aug 4, 2016 at 12:10 AM, Sandesh Hegde <sand...@datatorrent.com >>>> > >>>> wrote: >>>> >>>> Hello Team, >>>>> >>>>> This thread is to discuss the Named Checkpoint feature for Apex. ( >>>>> https://issues.apache.org/jira/browse/APEXCORE-498) >>>>> >>>>> Named checkpoints allow following workflow, >>>>> >>>>> 1. Users can trigger a checkpoint and give it a name >>>>> 2. Relaunch the application from the named checkpoint. >>>>> 3. These checkpoints survive the "purge of old checkpoints". >>>>> >>>>> Current idea is to add a new control tuple, NamedCheckPointTuple, which >>>>> contains the user specified name, it traverses the DAG and along the >>>>> >>>> way >>> >>>> necessary actions are taken. >>>>> >>>>> Please let me know your thoughts on this. >>>>> >>>>> Thanks >>>>> >>>>> >
