Right, this can work. Thanks for providing detailed queries... but there is an issue for me if i remove rows from that table, that can be used by other policies as central temporary repo.
so over the weekend i explored the code, and think this can put up the feature to access the expired events into pattern query. please notice there is a slight difference between using absence pattern and accessing expired events in pattern query. so I added this code change to use "....select expired events into..." in pattern query to give the required result here. I think that can be useful. Issue: https://github.com/wso2/siddhi/issues/568 PR: https://github.com/wso2/siddhi/pull/569 Thanks Jayesh On Sat, Sep 30, 2017 at 10:43 PM, Grainier Perera <[email protected]> wrote: > Hi Jay, > > Since you are maintaining a jobTable, you can use that to identify > completed and non-completed jobs in a given time frame. Refer to the below > snippet; > > @Plan:name('ExecutionPlan') > > >> @Import('subStream:1.0.0') >> define stream subStream (pid int, sid int, status string); > > >> @Import('pubStream:1.0.0') >> define stream pubStream (pid int, status string); > > >> @Export('jobsNotCompletedInTime:1.0.0') >> define stream jobsNotCompletedInTime (pid int, sid int); > > >> @Export('jobsCompletedInTime:1.0.0') >> define stream jobsCompletedInTime (pid int, sid int); > > >> @Export('foo:1.0.0') >> define stream foo (pid int, sid int); > > >> @From(eventtable='rdbms', jdbc.url='jdbc:mysql://localhost:3306/pubsub', >> username='root', password='root', driver.name='com.mysql.jdbc.Driver', >> table.name='jobTable') >> define table jobTable(pid int, sid int); > > >> from pubStream as p join jobTable as t >> on p.pid == t.pid >> select p.pid, t.sid >> insert into allPSJobs; > > > > -- this will emit an event after 10 sec once a publish event arrives > > from pubStream#window.time(10 sec) >> select * >> insert expired events into *expiredPublisherEvents*; > > > > -- if the pattern matches within 10 sec, that means the job completed >> within time > > from every (a=allPSJobs) -> s=subStream[pid==a.pid and sid == a.sid] >> within 10 sec >> select s.pid, s.sid >> insert into *jobsCompletedInTime*; > > >> -- delete completed jobs from the table, then what's left in the table >> are the uncompleted jobs >> from jobsCompletedInTime >> delete jobTable >> on jobTable.pid == pid and jobTable.sid == sid; > > >> -- so if we join expiredPublisherEvents with the jobTable, the result >> will be the jobs that are not completed within time > > from expiredPublisherEvents as ep join jobTable as t >> on ep.pid == t.pid >> select ep.pid, t.sid >> insert into *jobsNotCompletedInTime*; > > >> -- now delete those uncompleted jobs from the jobTable as well >> from jobsNotCompletedInTime >> delete jobTable >> on jobTable.pid == pid and jobTable.sid == sid; > > > > Hope that helped; > > Regards, > Grainier. > > > On Fri, Sep 29, 2017 at 11:17 AM, Jayesh Senjaliya <[email protected]> > wrote: > >> Hi Gobinath, >> >> Thanks for the suggestion about absence pattern, but we wont be able to >> upgrade to Siddhi 4 anytime soon. >> >> I am basically at the point where I can get all the relevant (subscribe) >> events that happened during the given interval of first arrival of publish >> events. >> >> here AllPublisher = all registered publisher-subscriber events ( for each >> publisher event ) >> >> from registeryTable as s join #publisher as p on p.pid == s.pid >> select p.pid, s.sid insert into AllPublisher; >> >> >> from every(a=AllPublisher) -> s=subscriber[pid == a.pid] >> within 1 sec >> select a.pid, s.sid >> insert into completed_jobs_in_1_sec; >> >> now, i need to find out those events from publisher (a) that dint had any >> match from s within 1 sec. >> >> which I was expecting to be available with " insert *expired events* >> into not_completed_jobs", but looks like expired events are only available >> when window is used. I am also looking at the code to see if i should add >> this. >> >> Thanks >> Jayesh >> >> >> >> >> >> >> >> >> >> On Wed, Sep 27, 2017 at 4:21 AM, Gobinath <[email protected]> wrote: >> >>> Hi, >>> >>> If you can use Siddhi 4 snapshot release, it can be done using the new >>> feature 'Absent Pattern' added to Siddhi 4. The query to detect the events >>> that do not match the condition within 10 seconds is given below: >>> >>> from every e1=publisher -> not subscriber[e1.pid == pid] for 10 sec >>> select e1.pid >>> insert into not_completed_jobs_in_time; >>> >>> The above query waits for 10 seconds from the arrival of every publisher >>> event and if there is no subscriber event with an id satisfying the >>> condition arrived within that waiting period, the id of the publisher event >>> will be inserted into the not_completed_jobs_in_time stream. >>> >>> I guess the official document for Siddhi 4 is under construction. So you >>> can find more details about absent pattern at [1] >>> >>> Still, Siddhi 4 is not production ready so I wonder whether you can use >>> this feature or not. >>> >>> [1] http://www.javahelps.com/2017/08/detect-absence-of-events-ws >>> o2-siddhi.html >>> >>> >>> >>> >>> On Tue, Sep 26, 2017 at 10:05 PM, Jayesh Senjaliya <[email protected]> >>> wrote: >>> >>>> Hi Grainier, >>>> >>>> ya, i came across that example page, but i think that does not work in >>>> my use-case which is as follow. >>>> >>>> i have a publish event followed by multiple subscribe event for the >>>> same publish job. >>>> now i want to catch if certain jobs (publish -> subscribe) has been >>>> finished with 10 sec. >>>> I have all the registered jobs in db table, which i use to gather all >>>> the required publish-subscribe job events. >>>> >>>> define table jobTable( pid string, sid string); >>>> define stream pubStream (pid int, status string); >>>> define stream subStream (pid int, sid int, status string); >>>> >>>> -- this will get all the publish-> subscribe jobs events as master list >>>> from pubStream as p join jobTable as t >>>> on p.pid == t.pid >>>> select p.pid, t.sid insert into allPSJobs; >>>> >>>> -- this is where i need to do intersection where if subStream event is >>>> seen within 2 sec then remove that from master list ( allPSJobs ) if not >>>> include that in not_completed_jobs_in_time >>>> >>>> from every ( a=allPSJobs ) -> s= subStream[sid == a.sid and pid==a.pid ] >>>> within 2 sec >>>> select s.pid, s.sid insert into completed_jobs_in_time; >>>> >>>> >>>> hope that make sense from what i am trying to do. >>>> >>>> Thanks >>>> Jayesh >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Sep 25, 2017 at 8:39 AM, Grainier Perera <[email protected]> >>>> wrote: >>>> >>>>> Hi Jay, >>>>> >>>>> You can try something similar to this to get non-matched events during >>>>> last 10 secs; You can find some documentation on this as well; link >>>>> <https://docs.wso2.com/display/CEP420/Sample+0111+-+Detecting+non-occurrences+with+Patterns> >>>>> >>>>> >>>>> >>>>>> define stream publisher (pid string, time string); >>>>>> define stream subscriber (pid string, sid string, time string); >>>>> >>>>> >>>>>> from publisher#window.time(10 sec) >>>>>> select * >>>>>> insert expired events into expired_publisher; >>>>> >>>>> >>>>>> from every pub=publisher -> sub=subscriber[pub.pid == pid] or >>>>>> exp=expired_publisher[pub.pid == pid] >>>>>> select pub.pid as pid, pub.time as time, sub.pid as subPid >>>>>> insert into filter_stream; >>>>> >>>>> >>>>>> from filter_stream [(subPid is null)] >>>>>> select pid, time >>>>>> insert into not_seen_in_last_10_sec_events; >>>>> >>>>> >>>>> Moreover, I didn't get what you meant by "also is there a way to >>>>> perform intersection of events based on grouping or time window ?" can you >>>>> please elaborate on this? >>>>> >>>>> Regards, >>>>> >>>>> On Mon, Sep 25, 2017 at 11:02 AM, Jayesh Senjaliya < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> is there a way to get events that didnt match within the given time >>>>>> frame. >>>>>> >>>>>> for example: >>>>>> >>>>>> define stream publisher (pid string, time string); >>>>>> define stream subscriber (pid string, sid string, time string); >>>>>> >>>>>> from every (e1=publisher) -> e2=subscriber[e1.pid == pid] >>>>>> within 10 sec >>>>>> select e1.pid, e2.sid >>>>>> insert into seen_in_last_10_sec_events; >>>>>> >>>>>> >>>>>> so if i have matching event above, i will see it in >>>>>> seen_in_last_10_sec_events, but is there a way to get all events or non >>>>>> matched events during that last 10 seconds from publisher or subscriber ? >>>>>> >>>>>> also is there a way to perform intersection of events based on >>>>>> grouping or time window ? >>>>>> >>>>>> >>>>>> Thanks >>>>>> Jay >>>>>> >>>>>> _______________________________________________ >>>>>> Dev mailing list >>>>>> [email protected] >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Grainier Perera >>>>> Senior Software Engineer >>>>> Mobile : +94716122384 <+94%2071%20612%202384> >>>>> WSO2 Inc. | http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> *Gobinath** Loganathan* >>> Graduate Student, >>> Electrical and Computer Engineering, >>> Western University. >>> Email : [email protected] >>> Blog : javahelps.com <http://www.javahelps.com/> >>> >>> >> >> > > > -- > Grainier Perera > Senior Software Engineer > Mobile : +94716122384 <+94%2071%20612%202384> > WSO2 Inc. | http://wso2.com > lean.enterprise.middleware >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
