Hi devs, Dian and me and our teammates have investigated this for a long time. We think consolidating Flink SQL and CEP is an exciting thing for Flink. It'll make SQL more powerful and give users the ability to easily and quickly build CEP applications. And I find Flink community has also talked about this idea before, such as the mailing list [1] and [2] and Fabian & Till's talk in Flink Forward 2016 [3].
I think THIS IS THE POINT to bring up this topic again. Because we already have pattern matching foundation in Flink CEP library, and Stream SQL is ready now and Calcite has partially supported pattern matching syntax! We also drafted a design doc about how to integrate SQL and CEP, and how to support CEP on Table API. https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it would be great to hear your response. What do others think? [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Add-CEP-library-to-Flink-td9743.html#a9787 [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790 [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin Regards, Jark Wu 2017-06-07 13:50 GMT+08:00 Dian Fu <dia...@apache.org>: > Hi everyone, > > Flink's CEP library is a great library for complex event processing, more > and more customers are expressing their interests in it. But it also has > some limitations that users usually have to write a lot of code even for a > very simple pattern match use case as it currently only supports the Java > API. > We have investigated some popular CEP products such as esper [1] and siddhi > [2] and found that most of these CEP products support SQL-like expressions > such as EPL to describe the match pattern. But these solutions also have > the drawbacks that the pattern match languages are not standard SQL, the > learn curve is steep for users and it's impossible to integrate them into > the Flink Table API & SQL. > We find that Oracle's CEP solution CQL [3] supports a new pattern > recognition clause match_recognize which is a pattern recognition clause > proposed in this paper [4]. It proposes a set of new syntaxes to define > match pattern in sql expression. Calcite already supports part of this > standard [5]. I think it will be of great value to support expressing > pattern recognition clause with match_recognize clause by integrating it > with Flink Table API & SQL and the Flink CEP library. Any thoughts? > > [1] http://www.espertech.com > [2] https://github.com/wso2/siddhi > [3] > https://docs.oracle.com/middleware/1213/eventprocessing/cql-reference/ > GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531 > [4] > http://web.cs.ucla.edu/classes/winter17/cs240B/notes/ > row-pattern-recogniton-11.pdf > [5] https://issues.apache.org/jira/browse/CALCITE-1570 > > Best Regards, > Dian >
