Re: [Dev] CEP over Out-of-Order Data Streams

[Miyuru Dayarathna]

Hi Suho,

Today's afternoon I merged the out-of-order handler (OOH) code to Siddhi
and created a pull request. It comprises of a simple test case [2
]
which is described in [3
].
I tested the code with DEBS 2015 Grand Challenge's taxi trips data set [1
] as well. I used the
small 2 million events data set in the tests. Since the events in the
original data set were in sorted order, I introduced a delay to certain
events in the event stream before sending them to the OOH (Shown in green
color highlighting in [4
]).
The results are listed in [4
].
The OOH reorders event streams in both example scenarios except for few
glitches. For 100 events case there was one unordered event output by OOH
(Highlighted in yellow color in [4
]).
I am investigating solutions for such unordered events output by OOH.
However, a similar situation has been described in Figure 2 of [3
]
as well. Another notable observation is that considerable amount of events
were buffered in the OOH while the extension was running. In certain cases
(as shown in [4
])
there were about 41 events buffered in OOH. Hence, both the test cases
ended up outputting lesser number of events than what is input to OOH
because the remaining set of events were buffered inside the OOH when the
test case terminated. This is expected behavior of a buffer based
out-of-order handler. Next, I will continue experiments with full data sets
of DEBS 2013 and 2015 data sets and observe the characteristics of OOH
buffer for different out-of-order scenarios.

[1] http://www.debs2015.org/call-grand-challenge.html

[2]
https://github.com/miyurud/siddhi/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java

[3]
https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing

[4]
https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing

-- 
Thanks,
Miyuru Dayarathna
Senior Technical Lead
Mobile: +94713527783
Blog: http://miyurublog.blogspot.com

On Wed, Sep 16, 2015 at 11:10 PM, Sriskandarajah Suhothayan 
wrote:

> Hi Miyuru
>
> Any update on this ?
>
> Suho
>
> On Tue, Sep 8, 2015 at 7:58 PM, Miyuru Dayarathna 
> wrote:
>
>> Hi Suho,
>>
>> Ok, I will work on that.
>>
>> --
>> Thanks,
>> Miyuru Dayarathna
>> Senior Technical Lead
>> Mobile: +94713527783
>> Blog: http://miyurublog.blogspot.com
>>
>> On Tue, Sep 8, 2015 at 5:38 PM, Sriskandarajah Suhothayan 
>> wrote:
>>
>>> Can you merge this code to siddhi and send a pull we will be able to add
>>> this to the release
>>>
>>> Suho
>>>
>>> On Tue, Sep 8, 2015 at 5:17 PM, Miyuru Dayarathna 
>>> wrote:
>>>
 Hi Suho,

 I have completed developing the K-Slack extension for Siddhi and I have
 hosted the code in [1 ]. I have
 tested the KSlackExtension class [2
 ]
 using a test scenario described in [3
 ]
 and I have verified the correctness of the implementation. The test case is
 accessible from [4
 ].
 We need to discuss whether we should keep the following Siddhi syntax or
 change it to some different version by removing the use of "window."
 construct. This code assumes that the very first field present in the input
 event as the timestamp used for reordering the events.

 @info(name = 'query1') from inputStream#window.ooh:kslack() select
 eventtt , price, volume insert into outputStream;

 I have assigned myself the Redmine issue [5
 ]. Next, I will move onto
 testing the performance of K-slack approach using the DEBS 2013 Grand
 Challenge data set. Also I 

Re: [Dev] CEP over Out-of-Order Data Streams

[Sriskandarajah Suhothayan]

Thanks for the pull request there are some comments on the code please fix
them

Thanks
Suho

On Thu, Sep 17, 2015 at 4:56 PM, Miyuru Dayarathna  wrote:

> Hi Suho,
>
> Today's afternoon I merged the out-of-order handler (OOH) code to Siddhi
> and created a pull request. It comprises of a simple test case [2
> ]
> which is described in [3
> ].
> I tested the code with DEBS 2015 Grand Challenge's taxi trips data set [1
> ] as well. I used the
> small 2 million events data set in the tests. Since the events in the
> original data set were in sorted order, I introduced a delay to certain
> events in the event stream before sending them to the OOH (Shown in green
> color highlighting in [4
> ]).
> The results are listed in [4
> ].
> The OOH reorders event streams in both example scenarios except for few
> glitches. For 100 events case there was one unordered event output by OOH
> (Highlighted in yellow color in [4
> ]).
> I am investigating solutions for such unordered events output by OOH.
> However, a similar situation has been described in Figure 2 of [3
> ]
> as well. Another notable observation is that considerable amount of events
> were buffered in the OOH while the extension was running. In certain cases
> (as shown in [4
> ])
> there were about 41 events buffered in OOH. Hence, both the test cases
> ended up outputting lesser number of events than what is input to OOH
> because the remaining set of events were buffered inside the OOH when the
> test case terminated. This is expected behavior of a buffer based
> out-of-order handler. Next, I will continue experiments with full data sets
> of DEBS 2013 and 2015 data sets and observe the characteristics of OOH
> buffer for different out-of-order scenarios.
>
> [1] http://www.debs2015.org/call-grand-challenge.html
>
> [2]
> https://github.com/miyurud/siddhi/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java
>
> [3]
> https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing
>
> [4]
> https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing
>
> --
> Thanks,
> Miyuru Dayarathna
> Senior Technical Lead
> Mobile: +94713527783
> Blog: http://miyurublog.blogspot.com
>
> On Wed, Sep 16, 2015 at 11:10 PM, Sriskandarajah Suhothayan  > wrote:
>
>> Hi Miyuru
>>
>> Any update on this ?
>>
>> Suho
>>
>> On Tue, Sep 8, 2015 at 7:58 PM, Miyuru Dayarathna 
>> wrote:
>>
>>> Hi Suho,
>>>
>>> Ok, I will work on that.
>>>
>>> --
>>> Thanks,
>>> Miyuru Dayarathna
>>> Senior Technical Lead
>>> Mobile: +94713527783
>>> Blog: http://miyurublog.blogspot.com
>>>
>>> On Tue, Sep 8, 2015 at 5:38 PM, Sriskandarajah Suhothayan >> > wrote:
>>>
 Can you merge this code to siddhi and send a pull we will be able to
 add this to the release

 Suho

 On Tue, Sep 8, 2015 at 5:17 PM, Miyuru Dayarathna 
 wrote:

> Hi Suho,
>
> I have completed developing the K-Slack extension for Siddhi and I
> have hosted the code in [1 ].
> I have tested the KSlackExtension class [2
> ]
> using a test scenario described in [3
> ]
> and I have verified the correctness of the implementation. The test case 
> is
> accessible from [4
> ].
> We need to discuss whether we should keep the following Siddhi syntax or
> change it to some different version by removing the use of "window."
> construct. This code assumes that the very first field present in the 
> input
> event as the timestamp used for reordering the events.
>
> @info(name = 

Re: [Dev] CEP over Out-of-Order Data Streams

[Miyuru Dayarathna]

Hi Suho,

I fixed all the issues mentioned during the code review and pushed the
modifications to GitHub.

-- 
Thanks,
Miyuru Dayarathna
Senior Technical Lead
Mobile: +94713527783
Blog: http://miyurublog.blogspot.com

On Thu, Sep 17, 2015 at 10:20 PM, Sriskandarajah Suhothayan 
wrote:

> Thanks for the pull request there are some comments on the code please fix
> them
>
> Thanks
> Suho
>
> On Thu, Sep 17, 2015 at 4:56 PM, Miyuru Dayarathna 
> wrote:
>
>> Hi Suho,
>>
>> Today's afternoon I merged the out-of-order handler (OOH) code to Siddhi
>> and created a pull request. It comprises of a simple test case [2
>> ]
>> which is described in [3
>> ].
>> I tested the code with DEBS 2015 Grand Challenge's taxi trips data set [1
>> ] as well. I used the
>> small 2 million events data set in the tests. Since the events in the
>> original data set were in sorted order, I introduced a delay to certain
>> events in the event stream before sending them to the OOH (Shown in green
>> color highlighting in [4
>> ]).
>> The results are listed in [4
>> ].
>> The OOH reorders event streams in both example scenarios except for few
>> glitches. For 100 events case there was one unordered event output by OOH
>> (Highlighted in yellow color in [4
>> ]).
>> I am investigating solutions for such unordered events output by OOH.
>> However, a similar situation has been described in Figure 2 of [3
>> ]
>> as well. Another notable observation is that considerable amount of events
>> were buffered in the OOH while the extension was running. In certain cases
>> (as shown in [4
>> ])
>> there were about 41 events buffered in OOH. Hence, both the test cases
>> ended up outputting lesser number of events than what is input to OOH
>> because the remaining set of events were buffered inside the OOH when the
>> test case terminated. This is expected behavior of a buffer based
>> out-of-order handler. Next, I will continue experiments with full data sets
>> of DEBS 2013 and 2015 data sets and observe the characteristics of OOH
>> buffer for different out-of-order scenarios.
>>
>> [1] http://www.debs2015.org/call-grand-challenge.html
>>
>> [2]
>> https://github.com/miyurud/siddhi/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java
>>
>> [3]
>> https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing
>>
>> [4]
>> https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing
>>
>> --
>> Thanks,
>> Miyuru Dayarathna
>> Senior Technical Lead
>> Mobile: +94713527783
>> Blog: http://miyurublog.blogspot.com
>>
>> On Wed, Sep 16, 2015 at 11:10 PM, Sriskandarajah Suhothayan <
>> s...@wso2.com> wrote:
>>
>>> Hi Miyuru
>>>
>>> Any update on this ?
>>>
>>> Suho
>>>
>>> On Tue, Sep 8, 2015 at 7:58 PM, Miyuru Dayarathna 
>>> wrote:
>>>
 Hi Suho,

 Ok, I will work on that.

 --
 Thanks,
 Miyuru Dayarathna
 Senior Technical Lead
 Mobile: +94713527783
 Blog: http://miyurublog.blogspot.com

 On Tue, Sep 8, 2015 at 5:38 PM, Sriskandarajah Suhothayan <
 s...@wso2.com> wrote:

> Can you merge this code to siddhi and send a pull we will be able to
> add this to the release
>
> Suho
>
> On Tue, Sep 8, 2015 at 5:17 PM, Miyuru Dayarathna 
> wrote:
>
>> Hi Suho,
>>
>> I have completed developing the K-Slack extension for Siddhi and I
>> have hosted the code in [1 ].
>> I have tested the KSlackExtension class [2
>> ]
>> using a test scenario described in [3
>> ]
>> and I have verified the correctness of the implementation. The test case 
>> is
>> accessible from [4
>> 

Re: [Dev] CEP over Out-of-Order Data Streams

[Sriskandarajah Suhothayan]

Hi Miyuru

Any update on this ?

Suho

On Tue, Sep 8, 2015 at 7:58 PM, Miyuru Dayarathna  wrote:

> Hi Suho,
>
> Ok, I will work on that.
>
> --
> Thanks,
> Miyuru Dayarathna
> Senior Technical Lead
> Mobile: +94713527783
> Blog: http://miyurublog.blogspot.com
>
> On Tue, Sep 8, 2015 at 5:38 PM, Sriskandarajah Suhothayan 
> wrote:
>
>> Can you merge this code to siddhi and send a pull we will be able to add
>> this to the release
>>
>> Suho
>>
>> On Tue, Sep 8, 2015 at 5:17 PM, Miyuru Dayarathna 
>> wrote:
>>
>>> Hi Suho,
>>>
>>> I have completed developing the K-Slack extension for Siddhi and I have
>>> hosted the code in [1 ]. I have
>>> tested the KSlackExtension class [2
>>> ]
>>> using a test scenario described in [3
>>> ]
>>> and I have verified the correctness of the implementation. The test case is
>>> accessible from [4
>>> ].
>>> We need to discuss whether we should keep the following Siddhi syntax or
>>> change it to some different version by removing the use of "window."
>>> construct. This code assumes that the very first field present in the input
>>> event as the timestamp used for reordering the events.
>>>
>>> @info(name = 'query1') from inputStream#window.ooh:kslack() select
>>> eventtt , price, volume insert into outputStream;
>>>
>>> I have assigned myself the Redmine issue [5
>>> ]. Next, I will move onto testing
>>> the performance of K-slack approach using the DEBS 2013 Grand Challenge
>>> data set. Also I will try out other approaches for Out-of-Order event
>>> processing which I have mentioned in the slides.
>>>
>>> [1] https://github.com/miyurud/Siddhi_Dev
>>>
>>> [2]
>>> https://github.com/miyurud/Siddhi_Dev/blob/master/modules/siddhi-extensions/ooh/src/main/java/org/wso2/siddhi/extension/ooh/KSlackExtension.java
>>>
>>> [3]
>>> https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing
>>>
>>> [4]
>>> https://github.com/miyurud/Siddhi_Dev/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java
>>>
>>> [5] https://redmine.wso2.com/issues/4032
>>>
>>> 
>>> --
>>> Thanks,
>>> Miyuru Dayarathna
>>> Senior Technical Lead
>>> Mobile: +94713527783
>>> Blog: http://miyurublog.blogspot.com
>>>
>>
>>
>>
>> --
>>
>> *S. Suhothayan*
>> Technical Lead & Team Lead of WSO2 Complex Event Processor
>> *WSO2 Inc. *http://wso2.com
>> * *
>> lean . enterprise . middleware
>>
>>
>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog:
>> http://suhothayan.blogspot.com/ twitter:
>> http://twitter.com/suhothayan  | linked-in:
>> http://lk.linkedin.com/in/suhothayan *
>>
>
>
>
> --
> Thanks,
> Miyuru Dayarathna
> Senior Technical Lead
> Mobile: +94713527783
> Blog: http://miyurublog.blogspot.com
>



-- 

*S. Suhothayan*
Technical Lead & Team Lead of WSO2 Complex Event Processor
*WSO2 Inc. *http://wso2.com
* *
lean . enterprise . middleware


*cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/
twitter: http://twitter.com/suhothayan
 | linked-in:
http://lk.linkedin.com/in/suhothayan *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] CEP over Out-of-Order Data Streams

[Sriskandarajah Suhothayan]

Can you merge this code to siddhi and send a pull we will be able to add
this to the release

Suho

On Tue, Sep 8, 2015 at 5:17 PM, Miyuru Dayarathna  wrote:

> Hi Suho,
>
> I have completed developing the K-Slack extension for Siddhi and I have
> hosted the code in [1 ]. I have
> tested the KSlackExtension class [2
> ]
> using a test scenario described in [3
> ]
> and I have verified the correctness of the implementation. The test case is
> accessible from [4
> ].
> We need to discuss whether we should keep the following Siddhi syntax or
> change it to some different version by removing the use of "window."
> construct. This code assumes that the very first field present in the input
> event as the timestamp used for reordering the events.
>
> @info(name = 'query1') from inputStream#window.ooh:kslack() select eventtt
> , price, volume insert into outputStream;
>
> I have assigned myself the Redmine issue [5
> ]. Next, I will move onto testing
> the performance of K-slack approach using the DEBS 2013 Grand Challenge
> data set. Also I will try out other approaches for Out-of-Order event
> processing which I have mentioned in the slides.
>
> [1] https://github.com/miyurud/Siddhi_Dev
>
> [2]
> https://github.com/miyurud/Siddhi_Dev/blob/master/modules/siddhi-extensions/ooh/src/main/java/org/wso2/siddhi/extension/ooh/KSlackExtension.java
>
> [3]
> https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing
>
> [4]
> https://github.com/miyurud/Siddhi_Dev/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java
>
> [5] https://redmine.wso2.com/issues/4032
>
> 
> --
> Thanks,
> Miyuru Dayarathna
> Senior Technical Lead
> Mobile: +94713527783
> Blog: http://miyurublog.blogspot.com
>



-- 

*S. Suhothayan*
Technical Lead & Team Lead of WSO2 Complex Event Processor
*WSO2 Inc. *http://wso2.com
* *
lean . enterprise . middleware


*cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/
twitter: http://twitter.com/suhothayan
 | linked-in:
http://lk.linkedin.com/in/suhothayan *
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev