Re: [Dev] Frequency extension for CEP - TopK Capacity and processing time relation
Hi Manoj, The Frequency implementation will not be continued further with the stream-lib library. But thank you for your interest. Anyway I have added the project to the below location, so that you can check it. https://svn.wso2.com/wso2/interns/2013/asokp/FrequencyExtension/ BR *Asok Aravinda Perera* Software Engineer WSO2, Inc.;http://wso2.com/ http://www.google.com/url?q=http%3A%2F%2Fwso2.com%2Fsa=Dsntz=1usg=AFQjCNGJuLRux6KkJwXKVUCYOtEsNCmIAQ lean.enterprise.middleware Mobile: +94722241032 On Mon, Oct 20, 2014 at 1:01 PM, Manoj Gunawardena man...@wso2.com wrote: Hi Asok, Can you send the svn link to the source. We also did lot of memory optimizations for disk backed window. would like to share our experience. Thanks On Sun, Oct 19, 2014 at 8:15 PM, Asok Perera as...@wso2.com wrote: Hi All, I ran several test runs for getting the correlation between 'event processing time' and 'TopK capacity' for Frequency implementation (transformer extension). I have put the stats for each test runs below (Graphs are event vs processing time). According to the results, it seems we need to decide the most optimal TopK capacity for this. Common tests specs, Input event stream has 2 attributes (symbol and price). Symbol will be the selected attribute for frequency calculation. Client - 50 events within a for loop without a sleep time in between (to get the behaviour for maximum in-event stream rate). Calculating algorithm runs in a synchronised method. *TOPK_CAPACITY = 1* *Test Results,* *Average event processing time = 700.02 ms* *Processing time for all 50 events = 35001 ms* *[image: Inline image 2]* *TOPK_CAPACITY = 1000* *Test Results,* *Average event processing time = 296.68 ms* *Processing time for all 50 events = 14834 ms* *[image: Inline image 3]* *TOPK_CAPACITY = 500* *Test Results,* *Average event processing time = 3.88 ms* *Processing time for all 50 events = 194 ms* *[image: Inline image 4]* *TOPK_CAPACITY = 100* *Test Results,* *Average event processing time = 0.98 ms* *Processing time for all 50 events = 49 ms* *[image: Inline image 5]* According to the results, there is significant correlation between event processing time and TopK capacity. (TopK capacity represents the number of *distinct* attribute values for frequency consideration). Please suggest a suitable TopK Capacity value for a production environment. Then we can continue more testings changing some other parameters to find out the relations and best possible values. BR *Asok Aravinda Perera* Software Engineer WSO2, Inc.;http://wso2.com/ http://www.google.com/url?q=http%3A%2F%2Fwso2.com%2Fsa=Dsntz=1usg=AFQjCNGJuLRux6KkJwXKVUCYOtEsNCmIAQ lean.enterprise.middleware Mobile: +94722241032 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Manoj Gunawardena Tech Lead WSO2, Inc.: http://wso2.com lean.enterprise.middleware Mobile : +94 77 2291643 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Frequency extension for CEP - TopK Capacity and processing time relation
Hi Asok, Can you send the svn link to the source. We also did lot of memory optimizations for disk backed window. would like to share our experience. Thanks On Sun, Oct 19, 2014 at 8:15 PM, Asok Perera as...@wso2.com wrote: Hi All, I ran several test runs for getting the correlation between 'event processing time' and 'TopK capacity' for Frequency implementation (transformer extension). I have put the stats for each test runs below (Graphs are event vs processing time). According to the results, it seems we need to decide the most optimal TopK capacity for this. Common tests specs, Input event stream has 2 attributes (symbol and price). Symbol will be the selected attribute for frequency calculation. Client - 50 events within a for loop without a sleep time in between (to get the behaviour for maximum in-event stream rate). Calculating algorithm runs in a synchronised method. *TOPK_CAPACITY = 1* *Test Results,* *Average event processing time = 700.02 ms* *Processing time for all 50 events = 35001 ms* *[image: Inline image 2]* *TOPK_CAPACITY = 1000* *Test Results,* *Average event processing time = 296.68 ms* *Processing time for all 50 events = 14834 ms* *[image: Inline image 3]* *TOPK_CAPACITY = 500* *Test Results,* *Average event processing time = 3.88 ms* *Processing time for all 50 events = 194 ms* *[image: Inline image 4]* *TOPK_CAPACITY = 100* *Test Results,* *Average event processing time = 0.98 ms* *Processing time for all 50 events = 49 ms* *[image: Inline image 5]* According to the results, there is significant correlation between event processing time and TopK capacity. (TopK capacity represents the number of *distinct* attribute values for frequency consideration). Please suggest a suitable TopK Capacity value for a production environment. Then we can continue more testings changing some other parameters to find out the relations and best possible values. BR *Asok Aravinda Perera* Software Engineer WSO2, Inc.;http://wso2.com/ http://www.google.com/url?q=http%3A%2F%2Fwso2.com%2Fsa=Dsntz=1usg=AFQjCNGJuLRux6KkJwXKVUCYOtEsNCmIAQ lean.enterprise.middleware Mobile: +94722241032 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev -- Manoj Gunawardena Tech Lead WSO2, Inc.: http://wso2.com lean.enterprise.middleware Mobile : +94 77 2291643 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Frequency extension for CEP - TopK Capacity and processing time relation
On Sun, Oct 19, 2014 at 8:15 PM, Asok Perera as...@wso2.com wrote: Hi All, I ran several test runs for getting the correlation between 'event processing time' and 'TopK capacity' for Frequency implementation (transformer extension). I have put the stats for each test runs below (Graphs are event vs processing time). According to the results, it seems we need to decide the most optimal TopK capacity for this. Common tests specs, Input event stream has 2 attributes (symbol and price). Symbol will be the selected attribute for frequency calculation. Client - 50 events within a for loop without a sleep time in between (to get the behaviour for maximum in-event stream rate). Calculating algorithm runs in a synchronised method. *TOPK_CAPACITY = 1* *Test Results,* *Average event processing time = 700.02 ms* *Processing time for all 50 events = 35001 ms* *[image: Inline image 2]* *TOPK_CAPACITY = 1000* *Test Results,* *Average event processing time = 296.68 ms* *Processing time for all 50 events = 14834 ms* *[image: Inline image 3]* *TOPK_CAPACITY = 500* *Test Results,* *Average event processing time = 3.88 ms* *Processing time for all 50 events = 194 ms* *[image: Inline image 4]* *TOPK_CAPACITY = 100* *Test Results,* *Average event processing time = 0.98 ms* *Processing time for all 50 events = 49 ms* *[image: Inline image 5]* According to the results, there is significant correlation between event processing time and TopK capacity. (TopK capacity represents the number of *distinct* attribute values for frequency consideration). Please suggest a suitable TopK Capacity value for a production environment. Make it configurable through siddhi query.. We can mention necessary recommended value through the documentation (or allow user to decide on that) Thanks, Mohan Then we can continue more testings changing some other parameters to find out the relations and best possible values. BR *Asok Aravinda Perera* Software Engineer WSO2, Inc.;http://wso2.com/ http://www.google.com/url?q=http%3A%2F%2Fwso2.com%2Fsa=Dsntz=1usg=AFQjCNGJuLRux6KkJwXKVUCYOtEsNCmIAQ lean.enterprise.middleware Mobile: +94722241032 -- *V. Mohanadarshan* *Software Engineer,* *Data Technologies Team,* *WSO2, Inc. http://wso2.com http://wso2.com * *lean.enterprise.middleware.* email: mo...@wso2.com phone:(+94) 771117673 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
