Hello everyone, We are happy to announce that the "tableOnCalcite" branch is finally ready to be merged. It essentially provides the existing functionality of the Table API, but now the translation happens through Apache Calcite. You can find the changes rebased on top of the current master in [1]. We have removed the prototype streaming Table API functionality, which will be added back once PR [2] is merged.
We'll go through the changes once more and, if no objections, we would like to go ahead and merge this. Cheers, -Vasia. [1]: https://github.com/vasia/flink/tree/merge-table [2]: https://github.com/apache/flink/pull/1770 On 15 January 2016 at 10:59, Fabian Hueske <fhue...@gmail.com> wrote: > Hi everybody, > > as previously announced, I pushed a feature branch called "tableOnCalcite" > to the Flink repository. > We will use this branch to work on FLINK-3221 and its sub-issues. > > Cheers, Fabian > > 2016-01-11 18:29 GMT+01:00 Fabian Hueske <fhue...@gmail.com>: > > > We haven't defined the StreamSQL syntax yet (and I think it will take > some > > time until we are at that point). > > So we are quite flexible with both featurs. > > > > Let's keep this opportunity in mind and coordinate when before making > > decisions about CEP or StreamSQL. > > > > Fabian > > > > 2016-01-11 17:29 GMT+01:00 Till Rohrmann <trohrm...@apache.org>: > > > >> First of all, it's a great design document. Looking forward having > stream > >> SQL in the foreseeable future :-) > >> > >> I think it is a good idea to consolidate stream SQL and CEP in the long > >> run. CEP's additional features compared to SQL boil down to pattern > >> detection. Once we have this, it should be only a question of defining > the > >> SQL syntax for event patterns in order to integrate CEP with stream SQL. > >> Oracle has already defined an extension [1] to detect patterns in a set > of > >> table rows. This or Esper's event processing language (EPL) [2] could > be a > >> good starting point. > >> > >> [1] https://docs.oracle.com/database/121/DWHSG/pattern.htm#DWHSG8959 > >> [2] http://www.espertech.com/esper/release-5.2.0/esper-reference/html/ > >> > >> Cheers, > >> Till > >> > >> On Mon, Jan 11, 2016 at 10:12 AM, Fabian Hueske <fhue...@gmail.com> > >> wrote: > >> > >> > Thanks for the feedback! > >> > > >> > We will start the SQL effort with putting the existing (batch) Table > >> API on > >> > top of Apache Calcite. > >> > From there we continue to add streaming support for the Table API > >> before we > >> > put a StreamSQL interface on top. > >> > > >> > Consolidating the efforts with the CEP library sounds like a good idea > >> to > >> > me. > >> > Maybe it can be nicely integrated with the streaming table API and > >> later as > >> > well with the StreamSQL interface (the StreamSQL dialect is not > defined > >> > yet). > >> > > >> > @Till: What do you think about adding CEP features to the Table API. > >> From > >> > the CEP design doc, it looks like we need to add a pattern matching > >> > operator in addition to the window features that we need to add for > >> > streaming Table API in any case. > >> > > >> > Best, Fabian > >> > > >> > 2016-01-11 4:03 GMT+01:00 Jiangsong (Hi) <hi.jiangs...@huawei.com>: > >> > > >> > > I suggest refering to Esper EPL[1], which is a SQL-standard language > >> > > extend to offering a cluster of window, pattern matching. EPL can > >> both > >> > > support Streaming SQL and CEP with one unified syntax. > >> > > > >> > > [1] > >> > > > >> > > >> > http://www.espertech.com/esper/release-5.2.0/esper-reference/pdf/esper_reference.pdf > >> > > (Chapter 5. EPL Reference: Clauses) > >> > > > >> > > > >> > > Regards > >> > > Song > >> > > > >> > > > >> > > -----邮件原件----- > >> > > 发件人: Chiwan Park [mailto:chiwanp...@apache.org] > >> > > 发送时间: 2016年1月11日 10:31 > >> > > 收件人: dev@flink.apache.org > >> > > 主题: Re: Effort to add SQL / StreamSQL to Flink > >> > > > >> > > We still don’t have a concensus about the streaming SQL and CEP > >> library > >> > on > >> > > Flink. Some people want to merge these two libraries. Maybe we have > to > >> > > discuss about this in mailing list. > >> > > > >> > > > On Jan 11, 2016, at 10:53 AM, Nick Dimiduk <ndimi...@gmail.com> > >> wrote: > >> > > > > >> > > > What's the relationship between the streaming SQL proposed here > and > >> > > > the CEP syntax proposed earlier in the week? > >> > > > > >> > > > On Sunday, January 10, 2016, Henry Saputra < > henry.sapu...@gmail.com > >> > > >> > > wrote: > >> > > > > >> > > >> Awesome! Thanks for the reply, Fabian. > >> > > >> > >> > > >> - Henry > >> > > >> > >> > > >> On Sunday, January 10, 2016, Fabian Hueske <fhue...@gmail.com > >> > > >> <javascript:;>> wrote: > >> > > >> > >> > > >>> Hi Henry, > >> > > >>> > >> > > >>> There is https://issues.apache.org/jira/browse/FLINK-2099 and a > >> few > >> > > >>> subissues. > >> > > >>> I'll reorganize these and add more issues for the tasks > described > >> in > >> > > >>> the design document in the next days. > >> > > >>> > >> > > >>> Thanks, Fabian > >> > > >>> > >> > > >>> 2016-01-10 2:45 GMT+01:00 Henry Saputra < > henry.sapu...@gmail.com > >> > > >> <javascript:;> > >> > > >>> <javascript:;>>: > >> > > >>> > >> > > >>>> HI Fabian, > >> > > >>>> > >> > > >>>> Have you created JIRA ticket to keep track of this new feature? > >> > > >>>> > >> > > >>>> - Henry > >> > > >>>> > >> > > >>>> On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske < > fhue...@gmail.com > >> > > >> <javascript:;> > >> > > >>> <javascript:;>> wrote: > >> > > >>>>> Hi everybody, > >> > > >>>>> > >> > > >>>>> in the last days, Timo and I refined the design document for > >> > > >>>>> adding a > >> > > >>>> SQL / > >> > > >>>>> StreamSQL interface on top of Flink that was started by > Stephan. > >> > > >>>>> > >> > > >>>>> The document proposes an architecture that is centered around > >> > > >>>>> Apache Calcite. Calcite is an Apache top-level project and > >> > > >>>>> includes a SQL > >> > > >>>> parser, > >> > > >>>>> a semantic validator for relational queries, and a rule- and > >> > > >> cost-based > >> > > >>>>> relational optimizer. Calcite is used by Apache Hive and > Apache > >> > > >>>>> Drill (among other projects). In a nutshell, the plan is to > >> > > >>>>> translate Table > >> > > >>> API > >> > > >>>>> and SQL queries into Calcite's relational expression trees, > >> > > >>>>> optimize > >> > > >>>> these > >> > > >>>>> trees, and translate them into DataSet and DataStream > >> programs.The > >> > > >>>> document > >> > > >>>>> breaks down the work into several tasks and subtasks. > >> > > >>>>> > >> > > >>>>> Please review the design document and comment. > >> > > >>>>> > >> > > >>>>> -- > > >> > > >>>>> > >> > > >>>> > >> > > >>> > >> > > >> > >> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjP > >> > > >> cp1h2TVqdI/edit?usp=sharing > >> > > >>>>> > >> > > >>>>> Unless there are major concerns with the design, Timo and I > want > >> > > >>>>> to > >> > > >>> start > >> > > >>>>> next week to move the current Table API on top of Apache > Calcite > >> > > >> (Task > >> > > >>> 1 > >> > > >>>> in > >> > > >>>>> the document). The goal of this task is to have the same > >> > > >> functionality > >> > > >>> as > >> > > >>>>> currently, but with Calcite in the translation process. This > is > >> a > >> > > >>>> blocking > >> > > >>>>> task that we hope to complete soon. Afterwards, we can > >> > > >>>>> independently > >> > > >>> work > >> > > >>>>> on different aspects such as extending the Table API, adding a > >> SQL > >> > > >>>>> interface (basically just a parser), integration with external > >> > > >>>>> data sources, better code generation, optimization rules, > >> > > >>>>> streaming > >> > > >> support > >> > > >>>> for > >> > > >>>>> the Table API, StreamSQL, etc.. > >> > > >>>>> > >> > > >>>>> Timo and I plan to work on a WIP branch to implement Task 1 > and > >> > > >>>>> merge > >> > > >>> it > >> > > >>>> to > >> > > >>>>> the master branch once the task is completed. Of course, > >> everybody > >> > > >>>>> is welcome to contribute to this effort. Please let us know > such > >> > > >>>>> that we > >> > > >>> can > >> > > >>>>> coordinate our efforts. > >> > > >>>>> > >> > > >>>>> Thanks, > >> > > >>>>> Fabian > >> > > > >> > > Regards, > >> > > Chiwan Park > >> > > > >> > > > >> > > > >> > > >> > > > > >
