Hey,
Thanks for the feedback guys:
@Max: You are right, this is not top priority to changes, I was just
mocking up some alternatives to try to make the state usage even simpler so
that the user can keep his current implementations and just add 1-2
annotations.
@Stephan, Robert: You are right that the checkpointed interface has some
advantages from that point of view. Maybe a way to go would be to separate
this signaling functionality (when the checkpoint is taken and maybe also
the commits) from the snapshotting itself. One advantage I see there is
that we would not need to have 3 different interfaces doing pretty much the
same thing (OperatorState - needed for partitioned state and different
backends/out-of-core, Checkpointed - needed for special actions after
checkpoints, Annotations - checkpointing simple fields natively).
What we could do is to modify the checkpointed interface, so it cannot
actually take a snapshot, but perform actions after the snapshot is taken,
and also on restore (and maybe include the commit functonality as well).
Checkpointed{
void onSnapshot(id, ts)
void onRestore(id, ts)
// void commit(id, ts)
}
Since all these special actions are best effort anyways I think this could
work. And now it is clear that OperatorStates and annotated fields would be
checkpointed, but we can still have actions on the checkpoints.
Would something like this make sense? I think it would be very good to
clarify and make the minimal necessary interfaces that don't necessarily
replicate functionality.
Cheers,
Gyula
Stephan Ewen <se...@apache.org> ezt írta (időpont: 2015. júl. 1., Sze,
10:22):
> +1 for adding the annotation, but not removing the interface
>
> Robert is right, the nice thing about the current interface is that you can
> use it to commit the state yourself to a database and simply return a key
> to where the state is stored. That is quite nice.
>
> On Wed, Jul 1, 2015 at 10:14 AM, Robert Metzger <rmetz...@apache.org>
> wrote:
>
> > I would certainly not replace the current Checkpointed interface by this
> > implementation.
> > The interface allows you to perform custom actions when creating a
> snapshot
> > or restoring state.
> >
> > We could add the annotation variant for simple cases like in your
> example.
> >
> >
> > On Wed, Jul 1, 2015 at 10:10 AM, Maximilian Michels <m...@apache.org>
> > wrote:
> >
> > > Hi Gyula,
> > >
> > > Looks like a neat feature you thought of; I like it. One problem I see
> is
> > > that with all the big changes and discussions in streaming, I wonder if
> > we
> > > should get other things right first. Another problem could be the lack
> of
> > > clarity of this implementation. State annotations can now be very
> easily
> > > placed anywhere in a class with the @Annotation. The Checkpointed
> > interface
> > > we have now is more explicit. From an end user perspective it might
> make
> > > sense to use annotations, from a developer perspective I'm not sure.
> > >
> > > Cheers,
> > > Max
> > >
> > >
> > >
> > > On Tue, Jun 30, 2015 at 1:44 PM, Hermann Gábor <reckone...@gmail.com>
> > > wrote:
> > >
> > > > Wow, this looks pretty concise. I really like it!
> > > >
> > > > On Mon, Jun 29, 2015 at 3:27 PM Gyula Fóra <gyf...@apache.org>
> wrote:
> > > >
> > > > > Hey all!
> > > > >
> > > > > Just to add something new to the end of the discussion list. After
> > some
> > > > > discussion with Seif, and Paris, I have added a commit that
> replaces
> > > the
> > > > > use of the Checkpointed interface with field annotations.
> > > > >
> > > > > This is probably the most lightweight state declaration so far and
> it
> > > > will
> > > > > probably work very well to replace the Checkpointed interface:
> > > > >
> > > > > public class StatefulMapper implements
> MapFunction<Integer,Integer> {
> > > > >
> > > > > @State
> > > > > int counter;
> > > > > @State
> > > > > Serializable state;
> > > > >
> > > > > Object notState
> > > > >
> > > > > public Integer map(Integer input)[
> > > > > counter++;
> > > > > //update other state
> > > > > /...
> > > > > }
> > > > > }
> > > > >
> > > > > What do you think?
> > > > > You can check it out here
> > > > > <https://github.com/gyfora/flink/commits/annotated_state>.
> > > > >
> > > > > Cheers,
> > > > > Gyula
> > > > >
> > > >
> > >
> >
>
