Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

[lgfmt]

Hi Sameer,
I just replied to the earlier post, but I will copy it here:
We also have the same requirement - we want to allow the user to change the 
matching patterns and have them take effect immediately. I'm wondering whether 
the proposed trigger DSL takes us one step closer:(I don't think it solves the 
problem) or we have to dynamically generate Flink job JAR files when the 
matching rules/patterns are changed and submit them to Flink.

I had thought about using a similar approach, but it is quite restrictive 
because you cannot use the power for Flink CEP engine with this approach. For 
example, I want to be able to use followedBy, next, notFollowedBy (in future) 
operators to detect the patterns and these matching patterns need to be 
user-cofigurable/dynamic/hot deployable. The simple attribute-based 
rules/patterns that you specified can be made dynamic as you mentioned but the 
rules/patterns that use not just the current event attributes, but also past 
events (e.g. followedBy) are much harder to make them dynamic without some help 
from Flink that implements the CEP operators.

- LF




  From: Sameer W 
 To: "user@flink.apache.org"  
 Sent: Tuesday, October 11, 2016 2:23 PM
 Subject: Re: What is the best way to load/add patterns dynamically (at 
runtime) with Flink?
   
I have used a JavaScript engine in my CEP to evaluate my patterns. Each event 
is a list of named attributes (HashMap like). And event is attached to a list 
of rules expressed as JavaScript code (See example below with one rule but I 
can match as many rules).  The rules are distributed over a connected stream 
which allow it to change over time. This is how I do it to keep my rules 
dynamic. If someone has a better way I would love to hear it as well.

private transient ScriptEngineManager factory = new ScriptEngineManager();
private transient ScriptEngine engine = factory.getEngineByName("JavaScript");
/*Inside open*/
factory = new ScriptEngineManager();

/*Close open*/

/*Inside my operator*/
engine = factory.getEngineByName("JavaScript");
engine.put("evt", value.f1); //value.f1 contains a JSON version of my HashMap 
of attributes
engine.eval(value.f2.rule); //f2.last contains the rule which is evaluated by 
the JavaScript Engine/*Sample JavaScript contained in the call - 
engine.eval(value.f2.rule); is shown below (not the "evt" variable in the 
JavaScript and the the preceding line - engine.put("evt", value.f1);
var evt=JSON.parse(evt);var result = evt.temperature>50 && evt.pressure<900
*/
boolean ret = (boolean)engine.get("result");
if(ret) /*Rule is Matched*/


> On Oct 11, 2016, at 5:01 PM, PedroMrChaves  wrote:
>
> Hello,
>
> I am new to Apache Flink and am trying to build a CEP using Flink's API. One
> of the requirements is the ability to add/change patterns at runtime for
> anomaly detection (maintaining the systems availability). Any Ideas of how
> could I do that?
>
> For instance, If I have a stream of security events (accesses,
> authentications ,etc.) and a pattern for detecting anomalies I would like to
> be able to change that pattern parameters, for instance instead of detecting
> the occurrence of events A->B->C I would like to change the condition on B
> to B’ in order to have a new rule. Moreover, I would like to be able to
> create new patterns dynamically as new use cases arise.
>
> Best Regards,
> Pedro Chaves
>
>
>
>
> --
> View this message in context: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at 
> Nabble.com.

   

Re: What is the best way to load/add patterns dynamically (at runtime) with Flink?

[lgfmt]

We also have the same requirement - we want to allow the user to change the 
matching patterns and have them take effect immediately. I'm wondering whether 
the proposed trigger DSL takes us one step closer:(I don't think it solves the 
problem) or we have to dynamically generate Flink job JAR files when the 
matching rules/patterns are changed and submit them to Flink.


- LF



  From: PedroMrChaves 
 To: user@flink.apache.org 
 Sent: Tuesday, October 11, 2016 2:01 PM
 Subject: What is the best way to load/add patterns dynamically (at runtime) 
with Flink?
   
Hello,

I am new to Apache Flink and am trying to build a CEP using Flink's API. One
of the requirements is the ability to add/change patterns at runtime for
anomaly detection (maintaining the systems availability). Any Ideas of how
could I do that?

For instance, If I have a stream of security events (accesses,
authentications ,etc.) and a pattern for detecting anomalies I would like to
be able to change that pattern parameters, for instance instead of detecting
the occurrence of events A->B->C I would like to change the condition on B
to B’ in order to have a new rule. Moreover, I would like to be able to
create new patterns dynamically as new use cases arise. 

Best Regards,
Pedro Chaves
 



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/What-is-the-best-way-to-load-add-patterns-dynamically-at-runtime-with-Flink-tp9461.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

   

Re: more complex patterns for CEP (was: CEP two transitions to the same state)

[lgfmt]

Thanks, Till. I will wait for your response.
- LF




  From: Till Rohrmann 
 To: user@flink.apache.org; lg...@yahoo.com 
 Sent: Tuesday, October 11, 2016 2:49 AM
 Subject: Re: more complex patterns for CEP (was: CEP two transitions to the 
same state)
   
The timeline is hard to predict to be honest. It depends a little bit on how 
fast the community can proceed with these things. At the moment I'm personally 
involved in other issues and, thus, cannot work on the CEP library. I hope to 
get back to it soon.
Cheers,Till
On Sat, Oct 8, 2016 at 12:42 AM,  wrote:

hi Till,
Thanks for the detailed response.
I'm looking forward to seeing these features implemented in Flink. Can anyone 
provide timelines for the 3 tickets that you mentioned in your response?  - LF




  From: Till Rohrmann 
 To: user@flink.apache.org 
 Sent: Tuesday, September 20, 2016 7:13 AM
 Subject: Re: more complex patterns for CEP (was: CEP two transitions to the 
same state)
  
Hi Frank,
thanks for sharing your analysis. It indeed pinpoints some of the current CEP 
library's shortcomings.
Let me address your points:
1. Lack of not operator
The functionality to express events which must not occur in a pattern is 
missing. We've currently a JIRA [1] which addresses exactly this. For the 
notFollowedBy operator, we should discard all patterns where we've seen a 
matching event for the not state. I think it could be implemented like a 
special terminal state where we prune the partial pattern.
For the notNext operator, we could think about keeping the event which has not 
matched the notNext state and return it as part of the fully matched pattern. 
Alternatively, we could simply forget about it once we've assured that it does 
not match.
2. Allow functions to access fields of previous events
This hasn't been implemented yet because it is a quite expensive operation. 
Before calling the filter function you always have to reconstruct the current 
partial pattern and then give it to the filter function. But I agree that the 
user should be allowed to use such a functionality (and then pay the price for 
it in terms of efficiency). Giving access to the partially matched fields via a 
Map would be a way to solve the problem on the API level.
I think that almost all functionality for this feature is already in place. We 
simply would have to check the filter condition whether they require access to 
previous events and then compute the partial pattern.
3. Support for recursive patterns
The underlying SharedBuffer implementation should allow recursive event 
patterns. Once we have support for branching CEP patterns [2] which allow to 
connect different states this should also be possible with some minor changes.
However, a more interesting way to specify recursive CEP patterns is to use 
regular expression syntax (Kleene star, bounded occurrences) to express 
recursive parts of a pattern. I think this makes specifying such a pattern 
easier and more intuitive for the user. We've also a JIRA issue to track the 
process there [3] and Ivan is already working on this.
If you want to get involved in Flink's CEP development, then feel free to take 
over any free JIRA issue or create one yourself :-)
[1] https://issues.apache.org/ jira/browse/FLINK-3320
[2] https://issues.apache.org/ jira/browse/FLINK-4641[3] 
https://issues.apache.org/ jira/browse/FLINK-3318
Cheers,Till
On Fri, Sep 16, 2016 at 10:04 PM, Frank Dekervel  wrote:

Hello,
i did some more analysis wrt the problem i'm facing and the flink CEP api.
In order to complete the problem i'm facing using flink CEP i would need 3 
additions to the API (i think). I tried to understand the NFA logic, and i 
think 2 of them should be doable without fundamental changes.
First is to add a "negative" pattern (notFollowedBy / notNext):
Reason is the flow below: i have a start and a termination event, and an 
optional "failure" event in between. i want all succesful termination events, 
so i want to express there should not be a failure event between the start and 
the termination event. Note that there is no "success" event in this case on 
which i could match.


To implement, upon checking whether a transition would be possible, one would 
first need to check if it was not already dead-ended by a notFollowedBy / 
notNext. This would add a bit of complexity to the logic (when seeing if a 
transition is valid for a state, first check if on this state there was not 
already a match made to an notFollowedBy/notNext state. in that case one would 
reject the match)
Second is to allow the filterfunction to inspect the partial match made, so one 
would be able to filter based on the already-matched event. Reason is the 
following (hypothetical) example where we would match arrivals of a trains in a 
station. We cannot keyBy train (because the "occupied" events of the station 
don't have train information), neither can we keyBy 

Re: Listening to timed-out patterns in Flink CEP

[lgfmt]

>>FLINK-3320 (CEP "not" operator) does not address this because again, how 
>>would the "not match" be triggered if no event at all occurs?
Good question. 
I'm not sure whether the following will work:
This could be done by creating a CEP matching pattern that uses both of 
"notNext" (or "notFollowedBy") and "within" constructs. Something like this:
Pattern pattern = Pattern.begin("first")
    .notNext("second")
    .within(Time.seconds(3));

I'm hoping Flink CEP experts (Till?) will comment on this.
Note: I have requested these negation patterns to be implemented in Flink CEP, 
but notNext/notFollowedBy are not yet implemented in Flink..


- LF

   


 From: David Koch 
 To: user@flink.apache.org; lg...@yahoo.com 
 Sent: Sunday, October 9, 2016 5:51 AM
 Subject: Re: Listening to timed-out patterns in Flink CEP
  
Hello,
Thank you for the explanation as well as the link to the other post. 
Interesting to learn about some of the open JIRAs.
Indeed, I was not using event time, but processing time. However, even when 
using event time I only get notified of timeouts upon subsequent events.
The link contains an example where I read   from a socket, wrap 
this in a custom "event" with timestamp, key the resultant stream by  and 
attempt to detect  instances no further than 3 seconds apart using CEP.
Apart from the fact that results are only printed when I close the socket 
(normal?) I don't observe any change in behaviour
So event-time/watermarks or not: SOME event has to occur for the timeout to be 
triggered.
FLINK-3320 (CEP "not" operator) does not address this because again, how would 
the "not match" be triggered if no event at all occurs?
On Sat, Oct 8, 2016 at 12:50 AM,  wrote:

The following is a better link:

http://mail-archives.apache. org/mod_mbox/flink-user/ 201609.mbox/%3CCAC27z% 
3DOTtv7USYUm82bE43- DkoGfVC4UAWD6uQwwRgTsE5be8g% 40mail.gmail.com%3E


- LF
 



  From: "lg...@yahoo.com" 
 To: "user@flink.apache.org"  
 Sent: Friday, October 7, 2016 3:36 PM
 Subject: Re: Listening to timed-out patterns in Flink CEP
   
Isn't the upcoming CEP negation (absence of an event) feature solve this issue?
See this discussion thread:http://mail-archives.apache. 
org/mod_mbox/flink-user/ 201609.mbox/%3CCAC27z%3DOD% 2BTq8twBw_ 
1YKni5sWAU3g1S9WDpJw0DUwgiG9YX 9Fg%40mail.gmail.com%3E

 //  Atul

  From: Till Rohrmann 
 To: user@flink.apache.org 
 Sent: Friday, October 7, 2016 12:58 AM
 Subject: Re: Listening to timed-out patterns in Flink CEP
  
Hi David,
in case of event time, the timeout will be detected when the first watermark 
exceeding the timeout value is received. Thus, it depends a little bit how you 
generate watermarks (e.g. periodically, watermark per event).
In case of processing time, the time is only updated whenever a new element 
arrives. Thus, if you have an element arriving 4 seconds after Event A, it 
should detect the timeout. If the next event arrives 20 seconds later, than you 
won't see the timeout until then.
In the case of processing time, we could think about registering timeout timers 
for processing time. However, I would highly recommend you to use event time, 
because with processing time, Flink cannot guarantee meaningful computations, 
because the events might arrive out of order.
Cheers,Till
On Thu, Oct 6, 2016 at 3:08 PM, David Koch  wrote:

Hello,
With Flink CEP, is there a way to actively listen to pattern matches that time 
out? I am under the impression that this is not possible.

In my case I partition a stream containing user web navigation by "userId" to 
look for sequences of Event A, followed by B within 4 seconds for each user.
I registered a PatternTimeoutFunction which assuming a non-match only fires 
upon the first event after the specified timeout. For example, given user X: 
Event A, 20 seconds later Event B (or any other type of event).
I'd rather have a notification fire directly upon the 4 second interval 
expiring since passive invalidation is not really applicable in my case.
How, if at all can this be achieved with Flink CEP?
Thanks,
David




   

   



   

Re: Listening to timed-out patterns in Flink CEP

[lgfmt]

The following is a better link:

http://mail-archives.apache.org/mod_mbox/flink-user/201609.mbox/%3CCAC27z%3DOTtv7USYUm82bE43-DkoGfVC4UAWD6uQwwRgTsE5be8g%40mail.gmail.com%3E


- LF
 



  From: "lg...@yahoo.com" 
 To: "user@flink.apache.org"  
 Sent: Friday, October 7, 2016 3:36 PM
 Subject: Re: Listening to timed-out patterns in Flink CEP
   
Isn't the upcoming CEP negation (absence of an event) feature solve this issue?
See this discussion 
thread:http://mail-archives.apache.org/mod_mbox/flink-user/201609.mbox/%3CCAC27z%3DOD%2BTq8twBw_1YKni5sWAU3g1S9WDpJw0DUwgiG9YX9Fg%40mail.gmail.com%3E

 //  Atul

  From: Till Rohrmann 
 To: user@flink.apache.org 
 Sent: Friday, October 7, 2016 12:58 AM
 Subject: Re: Listening to timed-out patterns in Flink CEP
  
Hi David,
in case of event time, the timeout will be detected when the first watermark 
exceeding the timeout value is received. Thus, it depends a little bit how you 
generate watermarks (e.g. periodically, watermark per event).
In case of processing time, the time is only updated whenever a new element 
arrives. Thus, if you have an element arriving 4 seconds after Event A, it 
should detect the timeout. If the next event arrives 20 seconds later, than you 
won't see the timeout until then.
In the case of processing time, we could think about registering timeout timers 
for processing time. However, I would highly recommend you to use event time, 
because with processing time, Flink cannot guarantee meaningful computations, 
because the events might arrive out of order.
Cheers,Till
On Thu, Oct 6, 2016 at 3:08 PM, David Koch  wrote:

Hello,
With Flink CEP, is there a way to actively listen to pattern matches that time 
out? I am under the impression that this is not possible.

In my case I partition a stream containing user web navigation by "userId" to 
look for sequences of Event A, followed by B within 4 seconds for each user.
I registered a PatternTimeoutFunction which assuming a non-match only fires 
upon the first event after the specified timeout. For example, given user X: 
Event A, 20 seconds later Event B (or any other type of event).
I'd rather have a notification fire directly upon the 4 second interval 
expiring since passive invalidation is not really applicable in my case.
How, if at all can this be achieved with Flink CEP?
Thanks,
David




   

   

Re: more complex patterns for CEP (was: CEP two transitions to the same state)

[lgfmt]

hi Till,
Thanks for the detailed response.
I'm looking forward to seeing these features implemented in Flink. Can anyone 
provide timelines for the 3 tickets that you mentioned in your response?  - LF




  From: Till Rohrmann 
 To: user@flink.apache.org 
 Sent: Tuesday, September 20, 2016 7:13 AM
 Subject: Re: more complex patterns for CEP (was: CEP two transitions to the 
same state)
   
Hi Frank,
thanks for sharing your analysis. It indeed pinpoints some of the current CEP 
library's shortcomings.
Let me address your points:
1. Lack of not operator
The functionality to express events which must not occur in a pattern is 
missing. We've currently a JIRA [1] which addresses exactly this. For the 
notFollowedBy operator, we should discard all patterns where we've seen a 
matching event for the not state. I think it could be implemented like a 
special terminal state where we prune the partial pattern.
For the notNext operator, we could think about keeping the event which has not 
matched the notNext state and return it as part of the fully matched pattern. 
Alternatively, we could simply forget about it once we've assured that it does 
not match.
2. Allow functions to access fields of previous events
This hasn't been implemented yet because it is a quite expensive operation. 
Before calling the filter function you always have to reconstruct the current 
partial pattern and then give it to the filter function. But I agree that the 
user should be allowed to use such a functionality (and then pay the price for 
it in terms of efficiency). Giving access to the partially matched fields via a 
Map would be a way to solve the problem on the API level.
I think that almost all functionality for this feature is already in place. We 
simply would have to check the filter condition whether they require access to 
previous events and then compute the partial pattern.
3. Support for recursive patterns
The underlying SharedBuffer implementation should allow recursive event 
patterns. Once we have support for branching CEP patterns [2] which allow to 
connect different states this should also be possible with some minor changes.
However, a more interesting way to specify recursive CEP patterns is to use 
regular expression syntax (Kleene star, bounded occurrences) to express 
recursive parts of a pattern. I think this makes specifying such a pattern 
easier and more intuitive for the user. We've also a JIRA issue to track the 
process there [3] and Ivan is already working on this.
If you want to get involved in Flink's CEP development, then feel free to take 
over any free JIRA issue or create one yourself :-)
[1] https://issues.apache.org/jira/browse/FLINK-3320
[2] https://issues.apache.org/jira/browse/FLINK-4641[3] 
https://issues.apache.org/jira/browse/FLINK-3318
Cheers,Till
On Fri, Sep 16, 2016 at 10:04 PM, Frank Dekervel  wrote:

Hello,
i did some more analysis wrt the problem i'm facing and the flink CEP api.
In order to complete the problem i'm facing using flink CEP i would need 3 
additions to the API (i think). I tried to understand the NFA logic, and i 
think 2 of them should be doable without fundamental changes.
First is to add a "negative" pattern (notFollowedBy / notNext):
Reason is the flow below: i have a start and a termination event, and an 
optional "failure" event in between. i want all succesful termination events, 
so i want to express there should not be a failure event between the start and 
the termination event. Note that there is no "success" event in this case on 
which i could match.


To implement, upon checking whether a transition would be possible, one would 
first need to check if it was not already dead-ended by a notFollowedBy / 
notNext. This would add a bit of complexity to the logic (when seeing if a 
transition is valid for a state, first check if on this state there was not 
already a match made to an notFollowedBy/notNext state. in that case one would 
reject the match)
Second is to allow the filterfunction to inspect the partial match made, so one 
would be able to filter based on the already-matched event. Reason is the 
following (hypothetical) example where we would match arrivals of a trains in a 
station. We cannot keyBy train (because the "occupied" events of the station 
don't have train information), neither can we keyBy station (as the start of 
the sequence is outside the station), so we need to add an additional condition 
for the second event: the train number must equal the train number of the first 
one. And in the third event, the station number should equal the station number 
of the second one.
I think this could be accomplished by overloading the where function with a 
second filterfunction variant that takes 2 parameters: the event considered + 
the partial match (as a Map with T the class of the event)


Third one is - i think - more difficult to accomplish, and that's more complex 
graphs i asked in my 

Re: Listening to timed-out patterns in Flink CEP

[lgfmt]

Isn't the upcoming CEP negation (absence of an event) feature solve this issue?
See this discussion 
thread:http://mail-archives.apache.org/mod_mbox/flink-user/201609.mbox/%3CCAC27z%3DOD%2BTq8twBw_1YKni5sWAU3g1S9WDpJw0DUwgiG9YX9Fg%40mail.gmail.com%3E

 //  Atul

  From: Till Rohrmann 
 To: user@flink.apache.org 
 Sent: Friday, October 7, 2016 12:58 AM
 Subject: Re: Listening to timed-out patterns in Flink CEP
   
Hi David,
in case of event time, the timeout will be detected when the first watermark 
exceeding the timeout value is received. Thus, it depends a little bit how you 
generate watermarks (e.g. periodically, watermark per event).
In case of processing time, the time is only updated whenever a new element 
arrives. Thus, if you have an element arriving 4 seconds after Event A, it 
should detect the timeout. If the next event arrives 20 seconds later, than you 
won't see the timeout until then.
In the case of processing time, we could think about registering timeout timers 
for processing time. However, I would highly recommend you to use event time, 
because with processing time, Flink cannot guarantee meaningful computations, 
because the events might arrive out of order.
Cheers,Till
On Thu, Oct 6, 2016 at 3:08 PM, David Koch  wrote:

Hello,
With Flink CEP, is there a way to actively listen to pattern matches that time 
out? I am under the impression that this is not possible.

In my case I partition a stream containing user web navigation by "userId" to 
look for sequences of Event A, followed by B within 4 seconds for each user.
I registered a PatternTimeoutFunction which assuming a non-match only fires 
upon the first event after the specified timeout. For example, given user X: 
Event A, 20 seconds later Event B (or any other type of event).
I'd rather have a notification fire directly upon the 4 second interval 
expiring since passive invalidation is not really applicable in my case.
How, if at all can this be achieved with Flink CEP?
Thanks,
David




   

Re: Regarding Late Elements

[lgfmt]

Not yet.
I'm hoping a Flink export on this mailing list will reply.

- LF



  From: vinay patil <vinay18.pa...@gmail.com>
 To: user@flink.apache.org 
 Sent: Monday, October 3, 2016 8:09 AM
 Subject: Re: Regarding Late Elements
   
Hi LF,
So did you manage to get the workaround for it ?

I am using a Custom Trigger which is similar to 1.0.3 with few changes
Regards,Vinay Patil
On Mon, Oct 3, 2016 at 10:02 AM, lgfmt [via Apache Flink User Mailing List 
archive.] <[hidden email]> wrote:

 We have the same requirement - we cannot discard any data even if it arrives 
late. 
- LF
 



  From: Vinay Patil <[hidden email]>
 To: [hidden email] 
 Sent: Sunday, October 2, 2016 8:21 PM
 Subject: Regarding Late Elements
  
Hi Guys,
Just wanted to get an idea on Why Flink decided to completely discard late 
elements in the latest version ?, this was not the case in 1.0.3


P.S In our case the data is critical so we cannot discard a single record even 
if it is late, I have written a custom trigger (as suggested by Aljoscha) to 
even accept late elements.


Regards,Vinay Patil


 
   If you reply to this email, your message will be added to the discussion 
below: http://apache-flink-user- mailing-list-archive.2336050. 
n4.nabble.com/Regarding-Late- Elements-tp9284p9292.html   To start a new topic 
under Apache Flink User Mailing List archive., email [hidden email] 
 To unsubscribe from Apache Flink User Mailing List archive., click here.
 NAML 

 
View this message in context: Re: Regarding Late Elements
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.


   

Re: Regarding Late Elements

[lgfmt]

We have the same requirement - we cannot discard any data even if it arrives 
late. 
- LF
 



  From: Vinay Patil 
 To: user@flink.apache.org 
 Sent: Sunday, October 2, 2016 8:21 PM
 Subject: Regarding Late Elements
   
Hi Guys,
Just wanted to get an idea on Why Flink decided to completely discard late 
elements in the latest version ?, this was not the case in 1.0.3


P.S In our case the data is critical so we cannot discard a single record even 
if it is late, I have written a custom trigger (as suggested by Aljoscha) to 
even accept late elements.


Regards,Vinay Patil

   

Re: more complex patterns for CEP - Negation (was: CEP two transitions to the same state)

[lgfmt]

Hi,


We are also looking for negation (absence of an event) functionality in Flink 
CEP. Something like notFollowedBy/notNext that detects the following patterns 
will be great additions to Flink CEP (other CEP frameworks support negation):1. 
Occurrence of an event (that matches specific criteria) followed by absence of 
an event (that matches specific criteria) followed by another event (that 
matches specific different criteria)2. Occurrence of an event (that matches 
specific criteria) followed by absence of an event (that matches specific 
criteria) for a specific period

Could this be implemented?
Thank you.

- LF
 
  From: Frank Dekervel 
 To: user@flink.apache.org 
 Sent: Friday, September 16, 2016 1:04 PM
 Subject: more complex patterns for CEP (was: CEP two transitions to the same 
state)
  
Hello,
i did some more analysis wrt the problem i'm facing and the flink CEP api.
In order to complete the problem i'm facing using flink CEP i would need 3 
additions to the API (i think). I tried to understand the NFA logic, and i 
think 2 of them should be doable without fundamental changes.
First is to add a "negative" pattern (notFollowedBy / notNext):
Reason is the flow below: i have a start and a termination event, and an 
optional "failure" event in between. i want all succesful termination events, 
so i want to express there should not be a failure event between the start and 
the termination event. Note that there is no "success" event in this case on 
which i could match.


To implement, upon checking whether a transition would be possible, one would 
first need to check if it was not already dead-ended by a notFollowedBy / 
notNext. This would add a bit of complexity to the logic (when seeing if a 
transition is valid for a state, first check if on this state there was not 
already a match made to an notFollowedBy/notNext state. in that case one would 
reject the match)
Second is to allow the filterfunction to inspect the partial match made, so one 
would be able to filter based on the already-matched event. Reason is the 
following (hypothetical) example where we would match arrivals of a trains in a 
station. We cannot keyBy train (because the "occupied" events of the station 
don't have train information), neither can we keyBy station (as the start of 
the sequence is outside the station), so we need to add an additional condition 
for the second event: the train number must equal the train number of the first 
one. And in the third event, the station number should equal the station number 
of the second one.
I think this could be accomplished by overloading the where function with a 
second filterfunction variant that takes 2 parameters: the event considered + 
the partial match (as a Map with T the class of the event)


Third one is - i think - more difficult to accomplish, and that's more complex 
graphs i asked in my original e-mail (eg two states having 2 transitions ending 
in the same state). The problem here is that it allows one to construct cyclic 
states, and the PatternStream takes a Map as input, which cannot 
express a state occuring twice, neither the order (which event was the first 
and which event was the second). In the problem i'm trying to solve cyclic 
states are not necessary, but i can imagine usecases exist.

I think the NFA implementation would already allow such scenario's but the 
nfacompiler and the CEP api would need changing.
I wonder if the problem i'm facing is exotic (so a custom CEP would be more 
logic) or it is just something that should be implemented in the flink CEP. I'm 
relatively new to CEP, so i cannot compare which other systems/implementations. 
I'd like to try implementing the changes myself (at least the first two) after 
taking some advice here ...
thanks!greetings,Frank





On Wed, Sep 14, 2016 at 5:22 PM, Frank Dekervel  wrote:

Hello,
I'm trying to model a FSM using the flink CEP patterns. However, there is 
something i can't figure out as all the documentation examples are linear 
(either you go to the single possible next state, either no match).
Suppose that two transitions lead from one state to two different states. I 
guess this is doable by just defining multiple followedBy/next on the same 
state.
But what about two different states that can end up in the same state (in the 
order / delivery example: suppose there are two different delivery methods, 
having a separate starting state but resulting in the same end state). It is 
possible to deduplicate the "delivered" state but this would lead to difficult 
to manage patterns when things get more complex.
Thanks!greetings,Frank