Quidem can run on top of any JDBC data source (you just need to invoke with a connection factory by implementing a simple SPI). But it requires queries to terminate (i.e. can’t handle streaming queries). So, if Flink SQL is were able to run queries on an EMP table, then I think it could be tested using Quidem.
> On Aug 2, 2018, at 6:27 AM, Fabian Hueske <[email protected]> wrote: > > Hi Julian, > > It would be great to use the same test suite. > > We have quite a few tests in Flink but they are not super well organized. > I would love to have more structure for at least some of the tests. > > I had a quick look at how Calcite runs its Quidem tests. > Not sure if this is a format that we could easily adopt to, but maybe its > possible to put a test data set, queries, and results in a more portable > format. > > Best, Fabian > > > > > > 2018-07-31 19:54 GMT+02:00 Julian Hyde <[email protected]>: > >> I’m delighted that Flink is getting full SQL support for MATCH_RECOGNIZE. >> >> Sounds like it might be challenging to share the implementation, but could >> we perhaps share the test suite? (I.e. a set of SQL queries and their >> expected results.) >> >> I added a simple test in https://github.com/julianhyde/calcite/commit/ >> ee460847643ec17544f310088affd99be4028bb6 <https://github.com/ >> julianhyde/calcite/commit/ee460847643ec17544f310088affd99be4028bb6> that >> could be extended. >> >> Julian >> >> >>> On Jul 31, 2018, at 8:07 AM, Fabian Hueske <[email protected]> wrote: >>> >>> Hi everyone, >>> >>> I'd like to share the plans for MATCH_RECOGNIZE support in Flink. >>> >>> Flink features a so-called CEP library for quite some time [1]. The CEP >>> features is a popular feature and frequently used. >>> In a nutshell, the library provides a domain-specific API to define event >>> patterns. The patterns are translated into a state machine and evaluated >> in >>> a streaming program. >>> >>> Even before, we learned about about MATCH_RECOGNIZE, Till (another Flink >>> committer) and I gave a few talks about unifying SQL and CEP [2]. >>> Hence, we were quite excited when we learned about MATCH_RECOGNIZE and >> even >>> more when it was added to Calcite. >>> Shortly after that, we got a PR [3] which translated the parsed >>> MATCH_RECOGNIZE clause into patterns of our CEP library. >>> However, we never really got to the point of merging that contribution, >>> mainly because there were some inconsistencies in the semantics of >>> MATCH_RECOGNIZE and Flink's CEP library. >>> >>> Recently, a Flink committers picked up this feature again, validated the >>> the semantics, and made a few corrections [4]. >>> The CEP library is now ready to support a subset of the MATCH_RECOGNIZE >>> features. >>> Unfortunately, MATCH_RECOGNIZE support won't make it into the upcoming >>> 1.6.0 release, but the plans are to add it for the 1.7.0 release. >>> >>> Regarding the idea of sharing parts of the evaluation logic. >>> Flink has runtime support for a subset of the MATCH_RECOGNIZE clause. >>> Unfortunately, I am not familiar with the internals of Flink's CEP >> library >>> and don't know how portable it is. >>> >>> Best, Fabian >>> >>> [1] >>> https://ci.apache.org/projects/flink/flink-docs- >> release-1.5/dev/libs/cep.html <https://ci.apache.org/ >> projects/flink/flink-docs-release-1.5/dev/libs/cep.html> >>> [2] >>> https://www.slideshare.net/tillrohrmann/streaming- >> analytics-cep-two-sides-of-the-same-coin <https://www.slideshare.net/ >> tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin> >>> [3] https://github.com/apache/flink/pull/4502 < >> https://github.com/apache/flink/pull/4502> >>> [4] https://issues.apache.org/jira/browse/FLINK-9593 < >> https://issues.apache.org/jira/browse/FLINK-9593> >>> >>> 2018-07-23 21:03 GMT+02:00 Sergey Nuyanzin <[email protected] <mailto: >> [email protected]>>: >>> >>>> looks exciting. >>>> If it is possible I would like to take a part of it however I'm not sure >>>> about this week (I could since August) >>>> >>>> On Mon, Jul 23, 2018 at 9:10 PM, Michael Mior <[email protected] >> <mailto:[email protected]>> wrote: >>>> >>>>> This does sound like my idea of fun, but unfortunately I won't have >>>>> the time to contribute in the near future. I'll keep this on my radar >>>>> though. I also shared this message with all the students in our >>>>> research group and I wouldn't be surprised if there was someone >>>>> willing to jump in. Thanks for keeping this moving Julian! >>>>> >>>>> -- >>>>> Michael Mior >>>>> [email protected] <mailto:[email protected]> >>>>> Le lun. 23 juil. 2018 à 13:54, Julian Hyde <[email protected] <mailto: >> [email protected]>> a écrit : >>>>>> >>>>>> For quite a while we have had partial support for MATCH_RECOGNIZE. We >>>>> support it in the parser and validator, but there is no runtime >>>>> implementation. It’s a shame, because MATCH_RECOGNIZE is an incredibly >>>>> powerful SQL feature for both traditional SQL (it’s in Oracle 12c) and >>>> for >>>>> continuous query (aka complex event processing - CEP). >>>>>> >>>>>> I figure it’s time to change that. My plan is to implement it >>>>> incrementally, getting simple queries working to start with, then allow >>>>> people to add more complex queries. >>>>>> >>>>>> In a dev branch [1], I’ve added a method Enumerables.match[2]. The >> idea >>>>> is that if you supply an Enumerable of input data, a finite state >> machine >>>>> to figure out when a sequence of rows makes a match (represented by a >>>>> transition function: (state, row) -> state), and a function to convert >> a >>>>> matched set of rows to a set of output rows. The match method is fairly >>>>> straightforward, and I almost have it finished. >>>>>> >>>>>> The complexity is in generating the finite state machine, emitter >>>>> function, and so forth. >>>>>> >>>>>> Can someone help me with this task? If your idea of fun is >> implementing >>>>> database algorithms, this is about as much fun as it gets. You learned >>>>> about finite state machines in college - this is your chance to >> actually >>>>> write one! >>>>>> >>>>>> This might be a good joint project with the Flink community. I know >>>>> Flink are thinking of implementing CEP, and the algorithm we write here >>>>> could be shared with Flink (for use via Flink SQL or via the Flink >> API). >>>>>> >>>>>> Julian >>>>>> >>>>>> [1] https://github.com/julianhyde/calcite/commits/1935-match- >> recognize >>>> < >>>>> https://github.com/julianhyde/calcite/commits/1935-match-recognize < >> https://github.com/julianhyde/calcite/commits/1935-match-recognize>> >>>>>> >>>>>> [2] https://github.com/julianhyde/calcite/commit/ < >> https://github.com/julianhyde/calcite/commit/> >>>>> 4dfaf1bbee718aa6694a8ce67d829c32d04c7e87#diff- >>>>> 8a97a64204db631471c563df7551f408R73 <https://github.com/ < >> https://github.com/> >>>>> julianhyde/calcite/commit/4dfaf1bbee718aa6694a8ce67d829c >> 32d04c7e87#diff- >>>>> 8a97a64204db631471c563df7551f408R73> >>>>> >>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Sergey >> >>
