Hi Konstantin, Thanks for your suggestion. For the first idea, I agree that adding a timestamp field and making users able to schedule a rule is a useful feature. This might not require too much implementation work and I believe it can be achieved in this FLIP.
As for the second idea, Rule is a concept that contains pattern and the two are different. I think we can consider the following example use cases: - while one pattern requires input data to be grouped by user id, another pattern might require data to be grouped by item id - users might want to process matched results differently, according to which pattern the result matches to. In these use cases, simply making patterns dynamically change is not enough. Therefore I introduced the concept "rule" to include pattern and all other functions around it, like key selector and the pattern process function, and by dynamically changing rules the example cases above can be supported. Thus maybe rule could be a better naming in the functions that this flip propose to achieve. Best regards, Yunfeng On Mon, Dec 13, 2021 at 3:45 PM Konstantin Knauf <kna...@apache.org> wrote: > Thanks, Yufeng, for starting this discussion. I think this will be a very > popular feature. I've seen a lot of users asking for this in the past. So, > generally big +1. > > I think we should have a rough idea on how to expose this feature in the > other APIs. > > Two ideas: > > 1. In order to make this more deterministic in case of reprocessing and > out-of-orderness, I am wondering if we can add a timestamp to each rule > that determines the start time from which the rule should be in effect. > This can be an event or a processing time depending on the characteristics > of the pipeline. The timestamp would default to Long.MIN_TIMESTAMP if not > provided, which means effectively immediately. This could also be a follow > up, if you think it will make the implementation too complicated initially. > > 2. I am wondering, if we should name Rule->DynamicPatternHolder or so and > CEP.rule-> CEP.dynamicPatterns instead (other classes correspondingly)? > Rule is quite ambiguous and DynamicPattern seems more descriptive to me. > > On Mon, Dec 13, 2021 at 4:30 AM Nicholas Jiang <nicholasji...@apache.org> > wrote: > > > Hi Martijn, > > > > IMO, in this FLIP, we only need to introduce the general design of the > > Table API/SQL level. As for the design details, you can create a new > FLIP. > > And do we need to take into account the support for Batch mode if you > > expand the MATCH_RECOGNIZE function? About the dynamic rule engine > design, > > do you have any comments? This core of the FLIP is about the multiple > rule > > and dynamic rule changing mechanism. > > > > Best, > > Nicholas Jiang > > > > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk >
