Hi Miyuru, Would it be possible to share the Siddhi queries that were run as well just to get an idea the type of queries that were run? (e.g. Window queries, Filter queries etc.)
Thanks, Lasantha On 22 April 2015 at 10:48, Miyuru Dayarathna <[email protected]> wrote: > Hi Srinath, > > I have shared the performance numbers of the Siddhi 3.0.0 version of the > EmailProcessing Benchmark on [1 > <https://docs.google.com/a/wso2.com/spreadsheets/d/1coow1_jyv50vwAuLIL92mCzn72iwOM9B8yxFgDHNS4Y/edit?usp=sharing>]. > As per the discussion we had during the meeting, I will port the Siddhi > queries to CEP+Storm implementation today and will update this email thread > with the performance numbers from my local WSO2 CEP + Storm cluster. > > [1] > https://docs.google.com/a/wso2.com/spreadsheets/d/1coow1_jyv50vwAuLIL92mCzn72iwOM9B8yxFgDHNS4Y/edit?usp=sharing > > Thanks, > Miyuru > > > EmailProcessing Benchmark Stats > <https://docs.google.com/a/wso2.com/spreadsheets/d/1coow1_jyv50vwAuLIL92mCzn72iwOM9B8yxFgDHNS4Y/edit?usp=drive_web> > > > On Tue, Apr 21, 2015 at 9:58 AM, Praneesha Chandrasiri <[email protected] > > wrote: > >> Hi Miyuru, >> >> +1. Thanks for the feedback. Will work with you to improve the docs. >> >> On Mon, Apr 20, 2015 at 11:18 AM, Miyuru Dayarathna <[email protected]> >> wrote: >> >>> Hi Praneesha, >>> >>> As I mentioned earlier I was able to run the WSO2 CEP sample 2001 after >>> going through the documentation of sample 2001 [1 >>> <https://docs.wso2.com/display/CEP400/Sample+2001+-+Simple+filter+query+with+Apache+Storm+deployment>]. >>> I had to seek support from Tishan, Lasantha during this process. There are >>> several places in the documentation which I think could be improved which I >>> have mentioned below. >>> >>> I would like if the documentation could provide some more details on >>> setting up a storm cluster (i.e., a local Storm cluster) or at least >>> provide some pointers. The statement "2. Apache Storm deployment." is not >>> informative. I think we need more description of what we mean by Apache >>> Storm deployment. For example, we can point the reader to some of the >>> existing tutorials on setting up a storm cluster such as [3 >>> <http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/> >>> ][4 >>> <https://storm.apache.org/documentation/Setting-up-development-environment.html>]. >>> The Storm's documentation [4 >>> <https://storm.apache.org/documentation/Setting-up-development-environment.html>] >>> on setting up Storm development environment provides only part of the >>> required information. For example, to get my local Storm cluster run >>> properly, I had to use the default Storm configuration information listed >>> in "defaults.yaml" [5 >>> <https://github.com/apache/storm/blob/master/conf/defaults.yaml>] >>> rather than using the configurations on "storm.yaml.example" [6 >>> <https://github.com/apache/storm/blob/master/conf/storm.yaml.example>]. >>> >>> After setting up a local Storm cluster, I used the following set of >>> commands on three different terminals on Ubuntu to get the sample 2001 >>> running. >>> >>> #(1): Thrift server >>> on "~/software/wso2cep-4.0.0-SNAPSHOT/samples/consumers/wso2-event" run >>> ant -Dhost=localhost -Dprotocol=thrift -Dsn=2001 >>> >>> #(2): cep-server >>> on "~/software/wso2cep-4.0.0-SNAPSHOT/bin" run ./wso2cep-samples.sh -sn >>> 2001 >>> >>> #(3): client >>> on "~/software/wso2cep-4.0.0-SNAPSHOT/samples/producers/analytics-stats" >>> run ant -Devents=10 -Dhost=localhost -Dport=7611 >>> >>> I did not find a particular difference due to the execution order of the >>> (1) thrift server and the (2) WSO2 CEP server. Either one could be started >>> first. >>> >>> Note that (1) and (3) can be executed just by running ant without any >>> parameters. However, the parameters provided to ant are very important in >>> (1) and (3) if we are providing any. For example, the port 7611 in (3) is >>> very important. This is the Thrift port on which the CEP server is >>> listening. The producer needs to send events to this port rather than 7661 >>> on which the producer listens. In a local deployment this may confuse the >>> reader since the events will be directly sent to the producer process >>> rather than getting directed through CEP server and Storm if the reader >>> uses -Dport=7661 accidentally. Furthermore, there is no information on use >>> of port 7611 in the current documentation. Hence, it becomes difficult for >>> first time users to figure out such parameters if they try them out. >>> >>> IMO the documentation should say that once the WSO2 CEP server starts >>> there should be a topology named "PreprocessStats[-1234]" appearing on the >>> Storm UI page. If the sample runs correctly, we should find non-zero >>> topology stats values of this topology listed on Storm UI. >>> >>> Minor comments on English mistakes and the content of CEP Storm >>> documentation >>> >>> 1) "Filters events from analyticsStats stream and insert results into >>> filteredStatStream."-->"insert results" needs to be changed to "inserts >>> results" >>> 2) On [2 >>> <https://docs.wso2.com/display/CEP400/Setting+up+CEP+Samples#SettingupCEPSamples-Prerequisites>], >>> "artefacts" needs to be changed to "artifacts" >>> 3) Under the thrid point of "Prerequisites" what do we mean by "wka" of >>> the following setting? >>> <parameter name="membershipScheme">wka</parameter> >>> 4) what do we mean by >>> To enable CEP to talk to Apache Storm add the Nimbus, Zookeeper, etc >>> information at <CEP_HOME>/repository/conf/cep/storm/storm.yaml file on both >>> nodes. ? "both nodes"? >>> >>> We can work on improving the documentation on at least some of these >>> aspects. >>> >>> Thanks, >>> Miyuru >>> >>> [1] >>> https://docs.wso2.com/display/CEP400/Sample+2001+-+Simple+filter+query+with+Apache+Storm+deployment >>> [2] >>> https://docs.wso2.com/display/CEP400/Setting+up+CEP+Samples#SettingupCEPSamples-Prerequisites >>> [3] >>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/ >>> [4] >>> https://storm.apache.org/documentation/Setting-up-development-environment.html >>> [5] https://github.com/apache/storm/blob/master/conf/defaults.yaml >>> [6] https://github.com/apache/storm/blob/master/conf/storm.yaml.example >>> >>> On Mon, Apr 20, 2015 at 10:11 AM, Praneesha Chandrasiri < >>> [email protected]> wrote: >>> >>>> Hi Miyuru/Iranga, >>>> >>>> Noted. Thanks for the feedback. >>>> >>>> Let's work on improving the doc. >>>> >>>> On Mon, Apr 20, 2015 at 9:17 AM, Miyuru Dayarathna <[email protected]> >>>> wrote: >>>> >>>>> Hi Srinath, >>>>> >>>>> Last Friday (16th April) I was able to run the sample 2001 with the >>>>> local Storm cluster setup on my laptop. I completed implementing a sample >>>>> Java application which implements the "Email Processing Benchmark" >>>>> scenario >>>>> described in [1 >>>>> <https://developer.ibm.com/streamsdev/wp-content/uploads/sites/15/2014/04/Streams-and-Storm-April-2014-Final.pdf>]. >>>>> At the moment I am porting that application to WSO2 CEP on Apache Storm >>>>> implementation. I have several suggestions for improving the >>>>> documentation [ >>>>> 2 >>>>> <https://docs.wso2.com/display/CEP400/Sample+2001+-+Simple+filter+query+with+Apache+Storm+deployment>] >>>>> on setting up and running the sample 2001, which I will send later via >>>>> this >>>>> email thread. >>>>> >>>>> Thanks, >>>>> Miyuru >>>>> >>>>> [1] https://developer.ibm.com/streamsdev/wp >>>>> -content/uploads/sites/15/2014/04/Streams-and-Storm-April-2014-Final. >>>>> pdf >>>>> [2] >>>>> https://docs.wso2.com/display/CEP400/Sample+2001+-+Simple+filter+query+with+Apache+Storm+deployment >>>>> >>>>> On Thu, Apr 9, 2015 at 11:58 AM, Miyuru Dayarathna <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> @ Iranga >>>>>> Thank you for sharing the information. They are useful. >>>>>> >>>>>> @ Srinath >>>>>> I will talk to Praneesha if I find anything that needs to be updated >>>>>> on the CEP documentation. >>>>>> >>>>>> Thanks, >>>>>> Miyuru >>>>>> >>>>>> >>>>>> On Thu, Apr 9, 2015 at 11:34 AM, Iranga Muthuthanthri < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> On Thu, Apr 9, 2015 at 11:23 AM, Srinath Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Can Miyuru update the Doc? Miyuru can u update doc while you are at >>>>>>>> it. >>>>>>>> >>>>>>> Documented some steps on the setup[1] hope it helps. >>>>>>> >>>>>>> [1] >>>>>>> https://docs.google.com/a/wso2.com/document/d/1l_qewhLiY2cPLoBcz9TXr2i-m583A2qkZwYcGj_lDfc/edit# >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> --Srinath >>>>>>>> >>>>>>>> On Thu, Apr 9, 2015 at 10:46 AM, Miyuru Dayarathna < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Tishan, >>>>>>>>> >>>>>>>>> Thank you for providing me the information. I will work on setting >>>>>>>>> up the environment based on these guidelines and will get in touch >>>>>>>>> with >>>>>>>>> you/Lasantha if I face any problems. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Miyuru >>>>>>>>> >>>>>>>>> On Thu, Apr 9, 2015 at 10:31 AM, Tishan Dahanayakage < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Miyuru, >>>>>>>>>> >>>>>>>>>> For starters try out this[1] sample. But please note >>>>>>>>>> that storm-deployment-config.xml is now part of event-processing.xml >>>>>>>>>> which >>>>>>>>>> is at <CARBON_HOME>/repository/conf. So steps may vary slightly. >>>>>>>>>> AFAIK >>>>>>>>>> separate effort is there to enrich storm documentation. You can >>>>>>>>>> contribute >>>>>>>>>> to that as well with your experience on setting this as a fresh >>>>>>>>>> user. :) >>>>>>>>>> Also when setting the local storm cluster follow installing >>>>>>>>>> storm release locally section on [2]. You need to have Apache >>>>>>>>>> Zookeeper >>>>>>>>>> running also to run Apache Storm. >>>>>>>>>> After you have a working sample you can use the same setup to >>>>>>>>>> deploy different query plans into storm and test. >>>>>>>>>> >>>>>>>>>> Feel free to reach out to myself or Lasantha if there is any >>>>>>>>>> problem. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://docs.wso2.com/display/CEP400/Sample+2001+-+Simple+filter+query+with+Apache+Storm+deployment >>>>>>>>>> [2] >>>>>>>>>> https://storm.apache.org/documentation/Setting-up-development-environment.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Tishan >>>>>>>>>> >>>>>>>>>> On Wed, Apr 8, 2015 at 8:35 PM, Miyuru Dayarathna < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Is there any setup instructions document for setting up the >>>>>>>>>>> software environment for carbon-event-processing project (In >>>>>>>>>>> particular to >>>>>>>>>>> get the following code running?). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://github.com/wso2/carbon-event-processing/tree/master/components/event-processor/org.wso2.carbon.event.processor.common/src/main/java/org/wso2/carbon/event/processor/common/storm/component >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Miyuru >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Apr 8, 2015 at 8:05 AM, Miyuru Dayarathna < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Srinath, >>>>>>>>>>>> >>>>>>>>>>>> Ok,I will work on finding a good benchmark for these >>>>>>>>>>>> performance tests. Also I will talk with Suho and some other team >>>>>>>>>>>> member >>>>>>>>>>>> from CEP team as you mentioned. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Miyuru >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Apr 8, 2015 at 7:58 AM, Srinath Perera < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Miyuru, can you try this out and run a bench mark. Please find >>>>>>>>>>>>> a good benchmark and respond to this thread so we can discuss. >>>>>>>>>>>>> >>>>>>>>>>>>> Suho, can someone from CEP point him to details and help. >>>>>>>>>>>>> >>>>>>>>>>>>> --Srinath >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> ============================ >>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Miyuru Dayarathna >>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>> Mobile: +94713527783 >>>>>>>>>>>> Blog: http://miyurublog.blogspot.com >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Thanks, >>>>>>>>>>> Miyuru Dayarathna >>>>>>>>>>> Senior Technical Lead >>>>>>>>>>> Mobile: +94713527783 >>>>>>>>>>> Blog: http://miyurublog.blogspot.com >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Tishan Dahanayakage >>>>>>>>>> Software Engineer >>>>>>>>>> WSO2, Inc. >>>>>>>>>> Mobile:+94 716481328 >>>>>>>>>> >>>>>>>>>> Disclaimer: This communication may contain privileged or other >>>>>>>>>> confidential information and is intended exclusively for the >>>>>>>>>> addressee/s. >>>>>>>>>> If you are not the intended recipient/s, or believe that you may have >>>>>>>>>> received this communication in error, please reply to the sender >>>>>>>>>> indicating >>>>>>>>>> that fact and delete the copy you received and in addition, you >>>>>>>>>> should not >>>>>>>>>> print, copy, re-transmit, disseminate, or otherwise use the >>>>>>>>>> information >>>>>>>>>> contained in this communication. Internet communications cannot be >>>>>>>>>> guaranteed to be timely, secure, error or virus-free. The sender >>>>>>>>>> does not >>>>>>>>>> accept liability for any errors or omissions. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks, >>>>>>>>> Miyuru Dayarathna >>>>>>>>> Senior Technical Lead >>>>>>>>> Mobile: +94713527783 >>>>>>>>> Blog: http://miyurublog.blogspot.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks & Regards >>>>>>> >>>>>>> Iranga Muthuthanthri >>>>>>> (M) -0777-255773 >>>>>>> Team Product Management >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Miyuru Dayarathna >>>>>> Senior Technical Lead >>>>>> Mobile: +94713527783 >>>>>> Blog: http://miyurublog.blogspot.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Miyuru Dayarathna >>>>> Senior Technical Lead >>>>> Mobile: +94713527783 >>>>> Blog: http://miyurublog.blogspot.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks & Best Regards, >>>> *Praneesha Chandrasiri* >>>> *Technical Writer* >>>> *WSO2 Inc. * >>>> *Mobile: +(94) 718156888 <%2B%2894%29%20718156888>* >>>> *E-mail: [email protected] <[email protected]>* >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Miyuru Dayarathna >>> Senior Technical Lead >>> Mobile: +94713527783 >>> Blog: http://miyurublog.blogspot.com >>> >> >> >> >> -- >> Thanks & Best Regards, >> *Praneesha Chandrasiri* >> *Technical Writer* >> *WSO2 Inc. * >> *Mobile: +(94) 718156888 <%2B%2894%29%20718156888>* >> *E-mail: [email protected] <[email protected]>* >> >> > > > -- > Thanks, > Miyuru Dayarathna > Senior Technical Lead > Mobile: +94713527783 > Blog: http://miyurublog.blogspot.com > -- *Lasantha Fernando* Software Engineer - Data Technologies Team WSO2 Inc. http://wso2.com email: [email protected] mobile: (+94) 71 5247551
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
