Re: CEP and Within Clause
Thank you- It is very clear now. Sameer On Tue, Aug 2, 2016 at 10:29 AM, Till Rohrmann <till.rohrm...@gmail.com> wrote: > The CEP operator maintains for each pattern a window length. This means > that every starting event will set its own timeout value. > > So if T=51 arrives in the 11th minute, then it depends whether the second > T=31 arrived sometime between the 1st and 11th minute. If that's the case, > then you should also see a second matching. > > Cheers, > Till > > On Tue, Aug 2, 2016 at 10:20 PM, Sameer W <sam...@axiomine.com> wrote: > >> Thanks Till, >> >> In that case if I have a pattern - >> First = T > 30 >> Followed By = T > 50 >> Within 10 minutes >> >> If I get the following sequence of events within 10 minutes >> T=31, T=51, T=31, T=51 >> >> I assume the alert will fire twice now. >> >> But what happens if the last T=51 arrives in the 11th minute. If the >> partially matched pattern is discarded after 10 minutes how will the system >> detect T=51. Or do you mean that that timer (for the within clause) is >> reset each time the patter T>30 matches. In that case it would fire! >> >> Thanks, >> Sameer >> >> On Tue, Aug 2, 2016 at 10:02 AM, Till Rohrmann <trohrm...@apache.org> >> wrote: >> >>> Hi Sameer, >>> >>> the within clause of CEP uses neither tumbling nor sliding windows. It >>> is more like a session window which is started whenever an element which >>> matches the starting condition arrives. As long as new events which fulfill >>> the pattern definition arrive within the length of the window, they will be >>> added. If the pattern should not be completed within the specified time >>> interval, the partially matched pattern will be discarded. If you've >>> specified a timeout handler, then the timeout handler is called with the >>> partial pattern. >>> >>> At the moment, there is no way to re-insert elements in the upstream. >>> Actually there is also no need for it because the CEP operator will detect >>> the alert patterns if there are two temperature readings > 150 within 6 >>> seconds. >>> >>> Cheers, >>> Till >>> >>> >>> >>> On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <aljos...@apache.org> >>> wrote: >>> >>>> +Till, looping him in directly, he probably missed this because he was >>>> away for a while. >>>> >>>> >>>> >>>> On Tue, 26 Jul 2016 at 18:21 Sameer W <sam...@axiomine.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> It looks like the WithIn clause of CEP uses Tumbling Windows. I could >>>>> get it to use Sliding windows by using an upstream pipeline which uses >>>>> Sliding Windows and produces repeating elements (in each sliding window) >>>>> and applying a Watermark assigner on the resulting stream with elements >>>>> duplicated. I wanted to use the "followedBy" pattern where there is a >>>>> strong need for sliding windows. >>>>> >>>>> Is there a plan to add sliding windows to the within clause at some >>>>> point? >>>>> >>>>> The PatternStream class's "select" and "flatSelect" have overloaded >>>>> versions which take PatternTimeOut variable. Is there a way to insert some >>>>> of those elements back to the front of the stream. Say I am trying to find >>>>> a pattern where two temperature readings >150 within 6 second window >>>>> should >>>>> raise an alert. If only one was found, can I insert that one back in the >>>>> front of the stream on that task node (for that window pane) so that I can >>>>> find a pattern match in the events occurring in the next 6 seconds. If I >>>>> can do that, I don't need sliding windows. Else I cannot avoid using them >>>>> for such scenarios. >>>>> >>>>> Thanks, >>>>> Sameer >>>>> >>>> >>> >> >
Re: CEP and Within Clause
The CEP operator maintains for each pattern a window length. This means that every starting event will set its own timeout value. So if T=51 arrives in the 11th minute, then it depends whether the second T=31 arrived sometime between the 1st and 11th minute. If that's the case, then you should also see a second matching. Cheers, Till On Tue, Aug 2, 2016 at 10:20 PM, Sameer W <sam...@axiomine.com> wrote: > Thanks Till, > > In that case if I have a pattern - > First = T > 30 > Followed By = T > 50 > Within 10 minutes > > If I get the following sequence of events within 10 minutes > T=31, T=51, T=31, T=51 > > I assume the alert will fire twice now. > > But what happens if the last T=51 arrives in the 11th minute. If the > partially matched pattern is discarded after 10 minutes how will the system > detect T=51. Or do you mean that that timer (for the within clause) is > reset each time the patter T>30 matches. In that case it would fire! > > Thanks, > Sameer > > On Tue, Aug 2, 2016 at 10:02 AM, Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi Sameer, >> >> the within clause of CEP uses neither tumbling nor sliding windows. It is >> more like a session window which is started whenever an element which >> matches the starting condition arrives. As long as new events which fulfill >> the pattern definition arrive within the length of the window, they will be >> added. If the pattern should not be completed within the specified time >> interval, the partially matched pattern will be discarded. If you've >> specified a timeout handler, then the timeout handler is called with the >> partial pattern. >> >> At the moment, there is no way to re-insert elements in the upstream. >> Actually there is also no need for it because the CEP operator will detect >> the alert patterns if there are two temperature readings > 150 within 6 >> seconds. >> >> Cheers, >> Till >> >> >> >> On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >>> +Till, looping him in directly, he probably missed this because he was >>> away for a while. >>> >>> >>> >>> On Tue, 26 Jul 2016 at 18:21 Sameer W <sam...@axiomine.com> wrote: >>> >>>> Hi, >>>> >>>> It looks like the WithIn clause of CEP uses Tumbling Windows. I could >>>> get it to use Sliding windows by using an upstream pipeline which uses >>>> Sliding Windows and produces repeating elements (in each sliding window) >>>> and applying a Watermark assigner on the resulting stream with elements >>>> duplicated. I wanted to use the "followedBy" pattern where there is a >>>> strong need for sliding windows. >>>> >>>> Is there a plan to add sliding windows to the within clause at some >>>> point? >>>> >>>> The PatternStream class's "select" and "flatSelect" have overloaded >>>> versions which take PatternTimeOut variable. Is there a way to insert some >>>> of those elements back to the front of the stream. Say I am trying to find >>>> a pattern where two temperature readings >150 within 6 second window should >>>> raise an alert. If only one was found, can I insert that one back in the >>>> front of the stream on that task node (for that window pane) so that I can >>>> find a pattern match in the events occurring in the next 6 seconds. If I >>>> can do that, I don't need sliding windows. Else I cannot avoid using them >>>> for such scenarios. >>>> >>>> Thanks, >>>> Sameer >>>> >>> >> >
Re: CEP and Within Clause
Thanks Till, In that case if I have a pattern - First = T > 30 Followed By = T > 50 Within 10 minutes If I get the following sequence of events within 10 minutes T=31, T=51, T=31, T=51 I assume the alert will fire twice now. But what happens if the last T=51 arrives in the 11th minute. If the partially matched pattern is discarded after 10 minutes how will the system detect T=51. Or do you mean that that timer (for the within clause) is reset each time the patter T>30 matches. In that case it would fire! Thanks, Sameer On Tue, Aug 2, 2016 at 10:02 AM, Till Rohrmann <trohrm...@apache.org> wrote: > Hi Sameer, > > the within clause of CEP uses neither tumbling nor sliding windows. It is > more like a session window which is started whenever an element which > matches the starting condition arrives. As long as new events which fulfill > the pattern definition arrive within the length of the window, they will be > added. If the pattern should not be completed within the specified time > interval, the partially matched pattern will be discarded. If you've > specified a timeout handler, then the timeout handler is called with the > partial pattern. > > At the moment, there is no way to re-insert elements in the upstream. > Actually there is also no need for it because the CEP operator will detect > the alert patterns if there are two temperature readings > 150 within 6 > seconds. > > Cheers, > Till > > > > On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > >> +Till, looping him in directly, he probably missed this because he was >> away for a while. >> >> >> >> On Tue, 26 Jul 2016 at 18:21 Sameer W <sam...@axiomine.com> wrote: >> >>> Hi, >>> >>> It looks like the WithIn clause of CEP uses Tumbling Windows. I could >>> get it to use Sliding windows by using an upstream pipeline which uses >>> Sliding Windows and produces repeating elements (in each sliding window) >>> and applying a Watermark assigner on the resulting stream with elements >>> duplicated. I wanted to use the "followedBy" pattern where there is a >>> strong need for sliding windows. >>> >>> Is there a plan to add sliding windows to the within clause at some >>> point? >>> >>> The PatternStream class's "select" and "flatSelect" have overloaded >>> versions which take PatternTimeOut variable. Is there a way to insert some >>> of those elements back to the front of the stream. Say I am trying to find >>> a pattern where two temperature readings >150 within 6 second window should >>> raise an alert. If only one was found, can I insert that one back in the >>> front of the stream on that task node (for that window pane) so that I can >>> find a pattern match in the events occurring in the next 6 seconds. If I >>> can do that, I don't need sliding windows. Else I cannot avoid using them >>> for such scenarios. >>> >>> Thanks, >>> Sameer >>> >> >
Re: CEP and Within Clause
Hi Sameer, the within clause of CEP uses neither tumbling nor sliding windows. It is more like a session window which is started whenever an element which matches the starting condition arrives. As long as new events which fulfill the pattern definition arrive within the length of the window, they will be added. If the pattern should not be completed within the specified time interval, the partially matched pattern will be discarded. If you've specified a timeout handler, then the timeout handler is called with the partial pattern. At the moment, there is no way to re-insert elements in the upstream. Actually there is also no need for it because the CEP operator will detect the alert patterns if there are two temperature readings > 150 within 6 seconds. Cheers, Till On Tue, Aug 2, 2016 at 5:12 AM, Aljoscha Krettek <aljos...@apache.org> wrote: > +Till, looping him in directly, he probably missed this because he was > away for a while. > > > > On Tue, 26 Jul 2016 at 18:21 Sameer W <sam...@axiomine.com> wrote: > >> Hi, >> >> It looks like the WithIn clause of CEP uses Tumbling Windows. I could get >> it to use Sliding windows by using an upstream pipeline which uses Sliding >> Windows and produces repeating elements (in each sliding window) and >> applying a Watermark assigner on the resulting stream with elements >> duplicated. I wanted to use the "followedBy" pattern where there is a >> strong need for sliding windows. >> >> Is there a plan to add sliding windows to the within clause at some >> point? >> >> The PatternStream class's "select" and "flatSelect" have overloaded >> versions which take PatternTimeOut variable. Is there a way to insert some >> of those elements back to the front of the stream. Say I am trying to find >> a pattern where two temperature readings >150 within 6 second window should >> raise an alert. If only one was found, can I insert that one back in the >> front of the stream on that task node (for that window pane) so that I can >> find a pattern match in the events occurring in the next 6 seconds. If I >> can do that, I don't need sliding windows. Else I cannot avoid using them >> for such scenarios. >> >> Thanks, >> Sameer >> >
Re: CEP and Within Clause
+Till, looping him in directly, he probably missed this because he was away for a while. On Tue, 26 Jul 2016 at 18:21 Sameer W <sam...@axiomine.com> wrote: > Hi, > > It looks like the WithIn clause of CEP uses Tumbling Windows. I could get > it to use Sliding windows by using an upstream pipeline which uses Sliding > Windows and produces repeating elements (in each sliding window) and > applying a Watermark assigner on the resulting stream with elements > duplicated. I wanted to use the "followedBy" pattern where there is a > strong need for sliding windows. > > Is there a plan to add sliding windows to the within clause at some point? > > The PatternStream class's "select" and "flatSelect" have overloaded > versions which take PatternTimeOut variable. Is there a way to insert some > of those elements back to the front of the stream. Say I am trying to find > a pattern where two temperature readings >150 within 6 second window should > raise an alert. If only one was found, can I insert that one back in the > front of the stream on that task node (for that window pane) so that I can > find a pattern match in the events occurring in the next 6 seconds. If I > can do that, I don't need sliding windows. Else I cannot avoid using them > for such scenarios. > > Thanks, > Sameer >
CEP and Within Clause
Hi, It looks like the WithIn clause of CEP uses Tumbling Windows. I could get it to use Sliding windows by using an upstream pipeline which uses Sliding Windows and produces repeating elements (in each sliding window) and applying a Watermark assigner on the resulting stream with elements duplicated. I wanted to use the "followedBy" pattern where there is a strong need for sliding windows. Is there a plan to add sliding windows to the within clause at some point? The PatternStream class's "select" and "flatSelect" have overloaded versions which take PatternTimeOut variable. Is there a way to insert some of those elements back to the front of the stream. Say I am trying to find a pattern where two temperature readings >150 within 6 second window should raise an alert. If only one was found, can I insert that one back in the front of the stream on that task node (for that window pane) so that I can find a pattern match in the events occurring in the next 6 seconds. If I can do that, I don't need sliding windows. Else I cannot avoid using them for such scenarios. Thanks, Sameer
