Looks super cool, Till!
Especially the section about the Patterns is great.
For the other parts, I was wondering about the overlap with the TableAPI
and the SQL efforts.
I was thinking that a first version could really focus on the Patterns and
make the assumption that they are always applied on a KeyedStream.
That way the effort would focus on the most important new addition, and we
could evaluate whether we could reuse the TableAPI for the other
grouping/windowing/etc parts.
Basically, have initially something like this:
Pattern<Event> pattern = Pattern.<Event>next("e1").where( (evt) -> evt.id
== 42 )
.followedBy("e2").where( (evt) -> evt.id == 1337 )
.within(Time.minutes(10))
KeyedStream<Event> ks = input.keyBy( (evt) -> evt.getId() );
CEP.pattern(ks, pattern).select( new PatternSelectFunction<Event>() { ... }
);
All other parts could still be constructed around that in the end.
Any thoughts?
Greetings,
Stepahn
On Fri, Jan 8, 2016 at 5:50 PM, Gordon Tai (戴資力) <tzuli...@gmail.com> wrote:
> A definite +1 for this feature, thanks for your effort Till!
> Really look forward to the POC foundation and would like to help contribute
> where-ever possible.
>
> Pattern matching along with event time support seems to be another major
> breakthrough for stream processing framework options currently on the
> table.
>
> At our company, we've been using Flink to implement pattern matching very
> similar to the use cases detailed in Till's design doc for adtech related
> applications. A comprehensive and expressive DSL for these applications
> will be fantastic.
>
> On Sat, Jan 9, 2016 at 12:36 AM, Ufuk Celebi <u...@apache.org> wrote:
>
> >
> > > On 08 Jan 2016, at 15:54, Till Rohrmann <trohrm...@apache.org> wrote:
> > >
> > > Hi everybody,
> > >
> > > recently we've seen an increased interest in complex event processing
> > (CEP)
> > > by Flink users. Even though most functionality is already there to
> solve
> > > many use cases it would still be helpful for most users to have an easy
> > to
> > > use library. Having such a library which allows to define complex event
> > > patterns would increase Flink's user range to the CEP community. Once
> > > having laid the foundation, I'm optimistic that people will quickly
> pick
> > it
> > > up and further extend it.
> > >
> > > The major contribution of this library would be to add an efficient
> > > non-deterministic finite automaton which can detect complex event
> > patterns.
> > > For everything else, Flink already has most of the functionality in
> > place.
> > >
> > > I've drafted a design document for the first version. Please review it
> > and
> > > comment:
> > >
> > >
> >
> https://docs.google.com/document/d/15iaBCZkNcpqSma_qrF0GUyobKV_JttEDVuhNd0Y1aAU/edit?usp=sharing
> >
> > Thanks for sharing, Till! I think that this will be a very valuable
> > addition to Flink. Looking forward to it. :-)
> >
> > – Ufuk
> >
> >
>
>
> --
> Tzu-Li (Gordon) Tai
> Data Engineer @ VMFive
> vmfive.com
>
