Hi everybody, as previously announced, I pushed a feature branch called "tableOnCalcite" to the Flink repository. We will use this branch to work on FLINK-3221 and its sub-issues.
Cheers, Fabian 2016-01-11 18:29 GMT+01:00 Fabian Hueske <fhue...@gmail.com>: > We haven't defined the StreamSQL syntax yet (and I think it will take some > time until we are at that point). > So we are quite flexible with both featurs. > > Let's keep this opportunity in mind and coordinate when before making > decisions about CEP or StreamSQL. > > Fabian > > 2016-01-11 17:29 GMT+01:00 Till Rohrmann <trohrm...@apache.org>: > >> First of all, it's a great design document. Looking forward having stream >> SQL in the foreseeable future :-) >> >> I think it is a good idea to consolidate stream SQL and CEP in the long >> run. CEP's additional features compared to SQL boil down to pattern >> detection. Once we have this, it should be only a question of defining the >> SQL syntax for event patterns in order to integrate CEP with stream SQL. >> Oracle has already defined an extension [1] to detect patterns in a set of >> table rows. This or Esper's event processing language (EPL) [2] could be a >> good starting point. >> >> [1] https://docs.oracle.com/database/121/DWHSG/pattern.htm#DWHSG8959 >> [2] http://www.espertech.com/esper/release-5.2.0/esper-reference/html/ >> >> Cheers, >> Till >> >> On Mon, Jan 11, 2016 at 10:12 AM, Fabian Hueske <fhue...@gmail.com> >> wrote: >> >> > Thanks for the feedback! >> > >> > We will start the SQL effort with putting the existing (batch) Table >> API on >> > top of Apache Calcite. >> > From there we continue to add streaming support for the Table API >> before we >> > put a StreamSQL interface on top. >> > >> > Consolidating the efforts with the CEP library sounds like a good idea >> to >> > me. >> > Maybe it can be nicely integrated with the streaming table API and >> later as >> > well with the StreamSQL interface (the StreamSQL dialect is not defined >> > yet). >> > >> > @Till: What do you think about adding CEP features to the Table API. >> From >> > the CEP design doc, it looks like we need to add a pattern matching >> > operator in addition to the window features that we need to add for >> > streaming Table API in any case. >> > >> > Best, Fabian >> > >> > 2016-01-11 4:03 GMT+01:00 Jiangsong (Hi) <hi.jiangs...@huawei.com>: >> > >> > > I suggest refering to Esper EPL[1], which is a SQL-standard language >> > > extend to offering a cluster of window, pattern matching. EPL can >> both >> > > support Streaming SQL and CEP with one unified syntax. >> > > >> > > [1] >> > > >> > >> http://www.espertech.com/esper/release-5.2.0/esper-reference/pdf/esper_reference.pdf >> > > (Chapter 5. EPL Reference: Clauses) >> > > >> > > >> > > Regards >> > > Song >> > > >> > > >> > > -----邮件原件----- >> > > 发件人: Chiwan Park [mailto:chiwanp...@apache.org] >> > > 发送时间: 2016年1月11日 10:31 >> > > 收件人: dev@flink.apache.org >> > > 主题: Re: Effort to add SQL / StreamSQL to Flink >> > > >> > > We still don’t have a concensus about the streaming SQL and CEP >> library >> > on >> > > Flink. Some people want to merge these two libraries. Maybe we have to >> > > discuss about this in mailing list. >> > > >> > > > On Jan 11, 2016, at 10:53 AM, Nick Dimiduk <ndimi...@gmail.com> >> wrote: >> > > > >> > > > What's the relationship between the streaming SQL proposed here and >> > > > the CEP syntax proposed earlier in the week? >> > > > >> > > > On Sunday, January 10, 2016, Henry Saputra <henry.sapu...@gmail.com >> > >> > > wrote: >> > > > >> > > >> Awesome! Thanks for the reply, Fabian. >> > > >> >> > > >> - Henry >> > > >> >> > > >> On Sunday, January 10, 2016, Fabian Hueske <fhue...@gmail.com >> > > >> <javascript:;>> wrote: >> > > >> >> > > >>> Hi Henry, >> > > >>> >> > > >>> There is https://issues.apache.org/jira/browse/FLINK-2099 and a >> few >> > > >>> subissues. >> > > >>> I'll reorganize these and add more issues for the tasks described >> in >> > > >>> the design document in the next days. >> > > >>> >> > > >>> Thanks, Fabian >> > > >>> >> > > >>> 2016-01-10 2:45 GMT+01:00 Henry Saputra <henry.sapu...@gmail.com >> > > >> <javascript:;> >> > > >>> <javascript:;>>: >> > > >>> >> > > >>>> HI Fabian, >> > > >>>> >> > > >>>> Have you created JIRA ticket to keep track of this new feature? >> > > >>>> >> > > >>>> - Henry >> > > >>>> >> > > >>>> On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske <fhue...@gmail.com >> > > >> <javascript:;> >> > > >>> <javascript:;>> wrote: >> > > >>>>> Hi everybody, >> > > >>>>> >> > > >>>>> in the last days, Timo and I refined the design document for >> > > >>>>> adding a >> > > >>>> SQL / >> > > >>>>> StreamSQL interface on top of Flink that was started by Stephan. >> > > >>>>> >> > > >>>>> The document proposes an architecture that is centered around >> > > >>>>> Apache Calcite. Calcite is an Apache top-level project and >> > > >>>>> includes a SQL >> > > >>>> parser, >> > > >>>>> a semantic validator for relational queries, and a rule- and >> > > >> cost-based >> > > >>>>> relational optimizer. Calcite is used by Apache Hive and Apache >> > > >>>>> Drill (among other projects). In a nutshell, the plan is to >> > > >>>>> translate Table >> > > >>> API >> > > >>>>> and SQL queries into Calcite's relational expression trees, >> > > >>>>> optimize >> > > >>>> these >> > > >>>>> trees, and translate them into DataSet and DataStream >> programs.The >> > > >>>> document >> > > >>>>> breaks down the work into several tasks and subtasks. >> > > >>>>> >> > > >>>>> Please review the design document and comment. >> > > >>>>> >> > > >>>>> -- > >> > > >>>>> >> > > >>>> >> > > >>> >> > > >> >> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjP >> > > >> cp1h2TVqdI/edit?usp=sharing >> > > >>>>> >> > > >>>>> Unless there are major concerns with the design, Timo and I want >> > > >>>>> to >> > > >>> start >> > > >>>>> next week to move the current Table API on top of Apache Calcite >> > > >> (Task >> > > >>> 1 >> > > >>>> in >> > > >>>>> the document). The goal of this task is to have the same >> > > >> functionality >> > > >>> as >> > > >>>>> currently, but with Calcite in the translation process. This is >> a >> > > >>>> blocking >> > > >>>>> task that we hope to complete soon. Afterwards, we can >> > > >>>>> independently >> > > >>> work >> > > >>>>> on different aspects such as extending the Table API, adding a >> SQL >> > > >>>>> interface (basically just a parser), integration with external >> > > >>>>> data sources, better code generation, optimization rules, >> > > >>>>> streaming >> > > >> support >> > > >>>> for >> > > >>>>> the Table API, StreamSQL, etc.. >> > > >>>>> >> > > >>>>> Timo and I plan to work on a WIP branch to implement Task 1 and >> > > >>>>> merge >> > > >>> it >> > > >>>> to >> > > >>>>> the master branch once the task is completed. Of course, >> everybody >> > > >>>>> is welcome to contribute to this effort. Please let us know such >> > > >>>>> that we >> > > >>> can >> > > >>>>> coordinate our efforts. >> > > >>>>> >> > > >>>>> Thanks, >> > > >>>>> Fabian >> > > >> > > Regards, >> > > Chiwan Park >> > > >> > > >> > > >> > >> > >
