[email protected] has been doing something very similar but using it support native Flink IO within Apache Beam within the company he works for. Note that the community had a discussion about runner specific extensions and is currently leaning[1] towards having support for them for internal use cases but not allowing those extensions to be part of Apache Beam publicly.
1: https://lists.apache.org/thread.html/38b796c4c49823cf946affdb1a457ddf1d142403803b9c6a32442057@%3Cdev.beam.apache.org%3E On Wed, Oct 17, 2018 at 1:36 PM Urban, Jaroslav <[email protected]> wrote: > Dear Co-Beamers, > > > > I am curious about the possibility of using Complex Event Processing (CEP) > package in Flink from Apache Beam with the Flink Runner. > > > > I have an architectural question. > > > > I know that Apache Beam and its reference model ( > https://beam.apache.org/documentation/runners/capability-matrix/) does > not have any kind of support for complex event processing (aka pattern > matching on streams of events and their attributes). There is merely a JIRA > ticket suggesting a future development in this direction ( > https://issues.apache.org/jira/browse/BEAM-3767). > > > > Does anyone know if there is currently some kind of “pass-through” > transform in Apache Beam that would make it possible to access the existing > CEP functionality in Flink. Or –for that matter- enable accessing > runner-specific features in general? > > > > What do you think? Any ideas? > > > > Best regards > > > > *Jaroslav Urban* > > Consultant (DWH, OpenSource, Cloud) > [email protected] > >
