OK, I see. Flink doesn't have support for JDBC yet. Would need to look into that.
2018-08-02 21:35 GMT+02:00 Julian Hyde <[email protected]>: > Quidem can run on top of any JDBC data source (you just need to invoke > with a connection factory by implementing a simple SPI). But it requires > queries to terminate (i.e. can’t handle streaming queries). So, if Flink > SQL is were able to run queries on an EMP table, then I think it could be > tested using Quidem. > > > On Aug 2, 2018, at 6:27 AM, Fabian Hueske <[email protected]> wrote: > > > > Hi Julian, > > > > It would be great to use the same test suite. > > > > We have quite a few tests in Flink but they are not super well organized. > > I would love to have more structure for at least some of the tests. > > > > I had a quick look at how Calcite runs its Quidem tests. > > Not sure if this is a format that we could easily adopt to, but maybe its > > possible to put a test data set, queries, and results in a more portable > > format. > > > > Best, Fabian > > > > > > > > > > > > 2018-07-31 19:54 GMT+02:00 Julian Hyde <[email protected]>: > > > >> I’m delighted that Flink is getting full SQL support for > MATCH_RECOGNIZE. > >> > >> Sounds like it might be challenging to share the implementation, but > could > >> we perhaps share the test suite? (I.e. a set of SQL queries and their > >> expected results.) > >> > >> I added a simple test in https://github.com/julianhyde/calcite/commit/ > >> ee460847643ec17544f310088affd99be4028bb6 <https://github.com/ > >> julianhyde/calcite/commit/ee460847643ec17544f310088affd99be4028bb6> > that > >> could be extended. > >> > >> Julian > >> > >> > >>> On Jul 31, 2018, at 8:07 AM, Fabian Hueske <[email protected]> wrote: > >>> > >>> Hi everyone, > >>> > >>> I'd like to share the plans for MATCH_RECOGNIZE support in Flink. > >>> > >>> Flink features a so-called CEP library for quite some time [1]. The CEP > >>> features is a popular feature and frequently used. > >>> In a nutshell, the library provides a domain-specific API to define > event > >>> patterns. The patterns are translated into a state machine and > evaluated > >> in > >>> a streaming program. > >>> > >>> Even before, we learned about about MATCH_RECOGNIZE, Till (another > Flink > >>> committer) and I gave a few talks about unifying SQL and CEP [2]. > >>> Hence, we were quite excited when we learned about MATCH_RECOGNIZE and > >> even > >>> more when it was added to Calcite. > >>> Shortly after that, we got a PR [3] which translated the parsed > >>> MATCH_RECOGNIZE clause into patterns of our CEP library. > >>> However, we never really got to the point of merging that contribution, > >>> mainly because there were some inconsistencies in the semantics of > >>> MATCH_RECOGNIZE and Flink's CEP library. > >>> > >>> Recently, a Flink committers picked up this feature again, validated > the > >>> the semantics, and made a few corrections [4]. > >>> The CEP library is now ready to support a subset of the MATCH_RECOGNIZE > >>> features. > >>> Unfortunately, MATCH_RECOGNIZE support won't make it into the upcoming > >>> 1.6.0 release, but the plans are to add it for the 1.7.0 release. > >>> > >>> Regarding the idea of sharing parts of the evaluation logic. > >>> Flink has runtime support for a subset of the MATCH_RECOGNIZE clause. > >>> Unfortunately, I am not familiar with the internals of Flink's CEP > >> library > >>> and don't know how portable it is. > >>> > >>> Best, Fabian > >>> > >>> [1] > >>> https://ci.apache.org/projects/flink/flink-docs- > >> release-1.5/dev/libs/cep.html <https://ci.apache.org/ > >> projects/flink/flink-docs-release-1.5/dev/libs/cep.html> > >>> [2] > >>> https://www.slideshare.net/tillrohrmann/streaming- > >> analytics-cep-two-sides-of-the-same-coin <https://www.slideshare.net/ > >> tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin> > >>> [3] https://github.com/apache/flink/pull/4502 < > >> https://github.com/apache/flink/pull/4502> > >>> [4] https://issues.apache.org/jira/browse/FLINK-9593 < > >> https://issues.apache.org/jira/browse/FLINK-9593> > >>> > >>> 2018-07-23 21:03 GMT+02:00 Sergey Nuyanzin <[email protected] > <mailto: > >> [email protected]>>: > >>> > >>>> looks exciting. > >>>> If it is possible I would like to take a part of it however I'm not > sure > >>>> about this week (I could since August) > >>>> > >>>> On Mon, Jul 23, 2018 at 9:10 PM, Michael Mior <[email protected] > >> <mailto:[email protected]>> wrote: > >>>> > >>>>> This does sound like my idea of fun, but unfortunately I won't have > >>>>> the time to contribute in the near future. I'll keep this on my radar > >>>>> though. I also shared this message with all the students in our > >>>>> research group and I wouldn't be surprised if there was someone > >>>>> willing to jump in. Thanks for keeping this moving Julian! > >>>>> > >>>>> -- > >>>>> Michael Mior > >>>>> [email protected] <mailto:[email protected]> > >>>>> Le lun. 23 juil. 2018 à 13:54, Julian Hyde <[email protected] > <mailto: > >> [email protected]>> a écrit : > >>>>>> > >>>>>> For quite a while we have had partial support for MATCH_RECOGNIZE. > We > >>>>> support it in the parser and validator, but there is no runtime > >>>>> implementation. It’s a shame, because MATCH_RECOGNIZE is an > incredibly > >>>>> powerful SQL feature for both traditional SQL (it’s in Oracle 12c) > and > >>>> for > >>>>> continuous query (aka complex event processing - CEP). > >>>>>> > >>>>>> I figure it’s time to change that. My plan is to implement it > >>>>> incrementally, getting simple queries working to start with, then > allow > >>>>> people to add more complex queries. > >>>>>> > >>>>>> In a dev branch [1], I’ve added a method Enumerables.match[2]. The > >> idea > >>>>> is that if you supply an Enumerable of input data, a finite state > >> machine > >>>>> to figure out when a sequence of rows makes a match (represented by a > >>>>> transition function: (state, row) -> state), and a function to > convert > >> a > >>>>> matched set of rows to a set of output rows. The match method is > fairly > >>>>> straightforward, and I almost have it finished. > >>>>>> > >>>>>> The complexity is in generating the finite state machine, emitter > >>>>> function, and so forth. > >>>>>> > >>>>>> Can someone help me with this task? If your idea of fun is > >> implementing > >>>>> database algorithms, this is about as much fun as it gets. You > learned > >>>>> about finite state machines in college - this is your chance to > >> actually > >>>>> write one! > >>>>>> > >>>>>> This might be a good joint project with the Flink community. I know > >>>>> Flink are thinking of implementing CEP, and the algorithm we write > here > >>>>> could be shared with Flink (for use via Flink SQL or via the Flink > >> API). > >>>>>> > >>>>>> Julian > >>>>>> > >>>>>> [1] https://github.com/julianhyde/calcite/commits/1935-match- > >> recognize > >>>> < > >>>>> https://github.com/julianhyde/calcite/commits/1935-match-recognize < > >> https://github.com/julianhyde/calcite/commits/1935-match-recognize>> > >>>>>> > >>>>>> [2] https://github.com/julianhyde/calcite/commit/ < > >> https://github.com/julianhyde/calcite/commit/> > >>>>> 4dfaf1bbee718aa6694a8ce67d829c32d04c7e87#diff- > >>>>> 8a97a64204db631471c563df7551f408R73 <https://github.com/ < > >> https://github.com/> > >>>>> julianhyde/calcite/commit/4dfaf1bbee718aa6694a8ce67d829c > >> 32d04c7e87#diff- > >>>>> 8a97a64204db631471c563df7551f408R73> > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Best regards, > >>>> Sergey > >> > >> > >
