Also note that there is a calculation interval in the siddhi time series regression function[1]. You maybe able get some insight for this from that as well.
[1] https://docs.wso2.com/display/CEP400/Regression On Tue, May 24, 2016 at 11:03 AM, Maheshakya Wijewardena < mahesha...@wso2.com> wrote: > Hi Mahesh, > > As we discussed offline, we can use similar mechanism to train linear > regression models, logistic regression models and k-means clustering models. > > It is very interesting that i have found that somethings that can make use >> of our work. In the cep 4.0 documentation there is a Custom Stream >> Processor Extention program [1]. There is a example of >> LinearRegressionStreamProcessor [1]. >> > > As we have to train predictive models with Spark, you can write wrappers > around regression/clustering models of Spark. Refer to Siddhi time series > regression source codes[1][2]. You can write a streaming linear regression > class for ML in a similar fashion by wrapping Spark mllib implementations. > You can use the methods "addEvent", "removeEvent", etc. (may have to be > changed according to requirements) for the similar purpose. You can > introduce trainLinearRegression/LogisticRegression/Kmeans which does a > similar thing as in createLinearRegression in those time series functions. > In the processData method you can use Spark mllib classes to actually train > models and return the model weights, evaluation metrics. So, converting > streams into RDDs and retrieving information from the trained models shall > happen in this method. > > In the stream processor extension example, you can retrieve those values > then use them to train new models with new batches. Weights/cluster centers > maybe passed as initialization parameters for the wrappers. > > Please note that we have to figure out the best siddhi extension type for > this process. In the siddhi query, we define batch size, type of algorithm > and number of features (there can be more). After batch size number of > events received, train a model and save parameters, return evaluation > metric. With the next batch, retrain the model initialized with previously > learned parameters. > > We also may need to test the same scenario with a moving window, but I > suspect that that approach may become so slow as a model is trained each > time an event is received. So, we may have to change the number of slots > the moving window moves at a time (eg: not one by one, but ten by ten). > > Once this is resolved, majority of the research part will be finished and > all we will be left to do is implementing wrappers around the 3 learning > algorithms we consider. > > Best regards. > > [1] > https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/RegressionCalculator.java > [2] > https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/SimpleLinearRegressionCalculator.java > > > On Sat, May 21, 2016 at 2:55 PM, Mahesh Dananjaya < > dananjayamah...@gmail.com> wrote: > >> Hi Maheshkya, >> shall we use [1] for our work? i am checking the possibility. >> BR, >> Mahesh. >> [1] >> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >> [2] >> https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength >> [3] >> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function >> >> On Sat, May 21, 2016 at 2:44 PM, Mahesh Dananjaya < >> dananjayamah...@gmail.com> wrote: >> >>> Hi Maheshakya, >>> It is very interesting that i have found that somethings that can make >>> use of our work. In the cep 4.0 documentation there is a Custom Stream >>> Processor Extention program [1]. There is a example of >>> LinearRegressionStreamProcessor [1] and also i saw >>> private int batchSize = 1000000000; i am going through this one. >>> Please check whether we can use. WIll there be any compatibility or >>> support issue? >>> regards, >>> Mahesh. >>> >>> >>> [1] >>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >>> >>> On Sat, May 21, 2016 at 11:52 AM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi maheshakya, >>>> anyway how can test any siddhi extention after write it without >>>> integrating it to cep.can you please explain me the procedure. i am >>>> referring to [1] [2] [3] [4]. thank you. >>>> BR, >>>> Mahesh. >>>> >>>> [1] https://docs.wso2.com/display/CEP310/Writing+Extensions+to+Siddhi >>>> [2] https://docs.wso2.com/display/CEP310/Writing+a+Custom+Function >>>> [3] https://docs.wso2.com/display/CEP310/Writing+a+Custom+Window >>>> [4] https://docs.wso2.com/display/CEP400/Writing+Extensions+to+Siddhi >>>> >>>> On Thu, May 19, 2016 at 12:08 PM, Mahesh Dananjaya < >>>> dananjayamah...@gmail.com> wrote: >>>> >>>>> Hi Maheshakya, >>>>> thank you for the feedback. I have add data-sets into repo. >>>>> data-sets/lr. I am all right with next week.Now i am writing some examples >>>>> to collect samples and build mini batches and run the algorithms on those >>>>> mini-batches. thank you. will add those into repo soon.I am still working >>>>> on that siddhi extention.i will let you know the progress. >>>>> BR, >>>>> mahesh. >>>>> >>>>> On Thu, May 19, 2016 at 11:10 AM, Maheshakya Wijewardena < >>>>> mahesha...@wso2.com> wrote: >>>>> >>>>>> Hi Mahesh, >>>>>> >>>>>> I've look into your code sample of streaming linear regression. Looks >>>>>> good to me, apart from few issues in coding practices which we can >>>>>> improve >>>>>> when you're doing the implementations in carbon-ml and during the code >>>>>> reviews. You are using a set of files as mini-batches of data, right? Can >>>>>> you also send us the datasets you've been using. I'd like to run this. >>>>>> >>>>>> does that cep problem is now all right that we were trying to fix. I >>>>>>> am still using those pre-build versions. If so i can merge with the >>>>>>> latest >>>>>>> one. >>>>>> >>>>>> >>>>>> I'll check this and let you know. >>>>>> >>>>>> Can we arrange a meeting (preferably in WSO2 offices) in next week >>>>>> with ML team members as well. Coding period begins on next Monday, so >>>>>> it's >>>>>> better to get overall feedback from others and discuss more about the >>>>>> project. Let me know convenient time slots for you. I'll arrange a >>>>>> meeting >>>>>> with ML team. >>>>>> >>>>>> Best regards. >>>>>> >>>>>> On Wed, May 18, 2016 at 9:53 AM, Mahesh Dananjaya < >>>>>> dananjayamah...@gmail.com> wrote: >>>>>> >>>>>>> Hi Maheshakya, >>>>>>> Ok. I will check it.you have sent me those relevant references and i >>>>>>> am working on that thing.thank you. does that cep problem is now all >>>>>>> right >>>>>>> that we were trying to fix. I am still using those pre-build versions. >>>>>>> If >>>>>>> so i can merge with the latest one.thanks. >>>>>>> BR, >>>>>>> Mahesh. >>>>>>> >>>>>>> On Wed, May 18, 2016 at 9:44 AM, Maheshakya Wijewardena < >>>>>>> mahesha...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi Mahesh, >>>>>>>> >>>>>>>> You don't actually have to implement anything in spark streaming. >>>>>>>> Try to understand how streaming data is handled in and the specifics >>>>>>>> of the >>>>>>>> underlying algorithms in streaming. >>>>>>>> What we want to do is having the similar algorithms that support >>>>>>>> CEP event streams with siddhi. >>>>>>>> >>>>>>>> Best regards. >>>>>>>> >>>>>>>> On Wed, May 18, 2016 at 9:38 AM, Mahesh Dananjaya < >>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Maheshakya, >>>>>>>>> Did you check the repo. I will add recent works today.And also i >>>>>>>>> was going through the Java docs related to spark streaming work. It >>>>>>>>> is with >>>>>>>>> that scala API. thank you. >>>>>>>>> regards, >>>>>>>>> Mahesh. >>>>>>>>> >>>>>>>>> On Tue, May 17, 2016 at 10:11 AM, Mahesh Dananjaya < >>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Maheshakya, >>>>>>>>>> I have gone through the Java Docs and run some of the Spark >>>>>>>>>> examples on spark shell which are paramount improtant for our work. >>>>>>>>>> Then i >>>>>>>>>> have been writing my codes to check the Linear regression, K means >>>>>>>>>> for >>>>>>>>>> streaming. please check my git repo [1]. I think now i have to ask >>>>>>>>>> on dev >>>>>>>>>> regarding the capturing event streams for our work. I will update the >>>>>>>>>> recent things on git. check the park-example directory for java. >>>>>>>>>> examples >>>>>>>>>> run on git shell is not included there. In my case i think i have to >>>>>>>>>> build >>>>>>>>>> mini batches from data streams that comes as individual samples. Now >>>>>>>>>> i am >>>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>>> streams.thank you. >>>>>>>>>> regards, >>>>>>>>>> Mahesh. >>>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>>> >>>>>>>>>> On Tue, May 17, 2016 at 10:10 AM, Mahesh Dananjaya < >>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>> I have gone through the Java Docs and run some of the Spark >>>>>>>>>>> examples on spark shell which are paramount improtant for our work. >>>>>>>>>>> Then i >>>>>>>>>>> have been writing my codes to check the Linear regression, K means >>>>>>>>>>> for >>>>>>>>>>> streaming. please check my git repo [1]. I think now i have to ask >>>>>>>>>>> on dev >>>>>>>>>>> regarding the capturing event streams for our work. I will update >>>>>>>>>>> the >>>>>>>>>>> recent things on git. check the park-example directory for java. >>>>>>>>>>> examples >>>>>>>>>>> run on git shell is not included there. In my case i think i have >>>>>>>>>>> to build >>>>>>>>>>> mini batches from data streams that comes as individual samples. >>>>>>>>>>> Now i am >>>>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>>>> streams.thank you. >>>>>>>>>>> regards, >>>>>>>>>>> Mahesh. >>>>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>>>> >>>>>>>>>>> On Mon, May 16, 2016 at 1:19 PM, Mahesh Dananjaya < >>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>> thank you. i will update the repo today.thank you.i changed the >>>>>>>>>>>> carbon ml siddhi extention and see how the changes are effecting. >>>>>>>>>>>> i will >>>>>>>>>>>> update the progress as soon as possible.thank you. i had some >>>>>>>>>>>> problem in >>>>>>>>>>>> spark mllib dependency. i was fixing that. >>>>>>>>>>>> regards, >>>>>>>>>>>> Mahesh. >>>>>>>>>>>> p.s: do i need to maintain a blog? >>>>>>>>>>>> >>>>>>>>>>>> On Mon, May 16, 2016 at 10:02 AM, Maheshakya Wijewardena < >>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry for replying late. >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for the update. I believe you have done some >>>>>>>>>>>>> implementations with with Spark MLLIb algorithms in streaming >>>>>>>>>>>>> fashion as we >>>>>>>>>>>>> have discussed. If so, can you please share your code in a Github >>>>>>>>>>>>> repo. >>>>>>>>>>>>> >>>>>>>>>>>>> Now i want to implements some machine learning algorithms with >>>>>>>>>>>>>> importing mllib and want to run within your code base >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> For the moment you can try out editing the same class >>>>>>>>>>>>> PredictStreamProcessor in the siddhi extension in carbon-ml. >>>>>>>>>>>>> Later we will >>>>>>>>>>>>> add this separately. You should be able to add >>>>>>>>>>>>> org.apache.spark.mllib. >>>>>>>>>>>>> classes to there. >>>>>>>>>>>>> >>>>>>>>>>>>> And i want to see how event streams are coming from cep. As i >>>>>>>>>>>>>> think it is not in a RDD format since it is arriving as the >>>>>>>>>>>>>> individual >>>>>>>>>>>>>> samples. I will send a email to dev asking about how to get the >>>>>>>>>>>>>> streams. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Please pay attention to length[1] and lengthbatch[1] inbuilt >>>>>>>>>>>>> windows in siddhi. What you need to write are functions similar >>>>>>>>>>>>> to a custom >>>>>>>>>>>>> aggregate function[2]. >>>>>>>>>>>>> When you send the email to dev list, explain your requirement. >>>>>>>>>>>>> You need to get a set of event with from a stream with a >>>>>>>>>>>>> specified window >>>>>>>>>>>>> size (number of events). Then build a model within that function. >>>>>>>>>>>>> You also >>>>>>>>>>>>> need to retain the data (learned weights, cluster centers, etc.) >>>>>>>>>>>>> from the >>>>>>>>>>>>> previous window to use in the current window. Ask what can be the >>>>>>>>>>>>> most >>>>>>>>>>>>> suitable option for this among the set of siddhi extensions given. >>>>>>>>>>>>> >>>>>>>>>>>>> Best regards. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength >>>>>>>>>>>>> [2] >>>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 11, 2016 at 1:43 PM, Mahesh Dananjaya < >>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>>> From: Mahesh Dananjaya <dananjayamah...@gmail.com> >>>>>>>>>>>>>> Date: Wed, May 11, 2016 at 1:43 PM >>>>>>>>>>>>>> Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive analytic >>>>>>>>>>>>>> with online data for WSO2 Machine Learner >>>>>>>>>>>>>> To: Maheshakya Wijewardena <mahesha...@wso2.com> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>> sorry for not updating. I did what you wanted me to do. I >>>>>>>>>>>>>> checked the code base and train functions. I went through those >>>>>>>>>>>>>> java docs. >>>>>>>>>>>>>> I went through the carbon-ml current implementation of LG and >>>>>>>>>>>>>> K-Mean. And i >>>>>>>>>>>>>> had Apache Spark and i tried with several examples. Now i want to >>>>>>>>>>>>>> implements some machine learning algorithms with importing mllib >>>>>>>>>>>>>> and want >>>>>>>>>>>>>> to run within your code base. Can you help me with that. >>>>>>>>>>>>>> And i want to see how event streams are coming from cep. As i >>>>>>>>>>>>>> think it is not in a RDD format since it is arriving as the >>>>>>>>>>>>>> individual >>>>>>>>>>>>>> samples. I will send a email to dev asking about how to get the >>>>>>>>>>>>>> streams. I >>>>>>>>>>>>>> debugged many of those functions in the code base. So need >>>>>>>>>>>>>> further >>>>>>>>>>>>>> instructions to proceed.thank you. >>>>>>>>>>>>>> regards, >>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, May 11, 2016 at 10:32 AM, Maheshakya Wijewardena < >>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Any update on your progress? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, May 4, 2016 at 8:35 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> is that "Put break points in train methods in Linear >>>>>>>>>>>>>>>>> Regression class" means the spark/algorithms/ >>>>>>>>>>>>>>>>> LinearRegrassion.java class >>>>>>>>>>>>>>>>> in the org.wso2.carbon.ml.core? is that the correct file? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, this is the correct place. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You can refer to spark programming guide[1][2] as well as >>>>>>>>>>>>>>>> our ML code base when you try those algorithms out. Please try >>>>>>>>>>>>>>>> to do rough >>>>>>>>>>>>>>>> implementations of the streaming versions of linear >>>>>>>>>>>>>>>> regression, logistic >>>>>>>>>>>>>>>> regression and k-means clustering as we have discussed in the >>>>>>>>>>>>>>>> proposal in >>>>>>>>>>>>>>>> plain Java. It's better if you can create a git repo and share >>>>>>>>>>>>>>>> your code >>>>>>>>>>>>>>>> once you have made some progress. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Were you able debug and understand the flow of the ML >>>>>>>>>>>>>>>> siddhi extension? I hope you haven't encountered more errors >>>>>>>>>>>>>>>> after >>>>>>>>>>>>>>>> switching the released version of CEP. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is this Friday okay for you? Afternoon at 2:00 pm? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>> http://spark.apache.org/docs/latest/programming-guide.html >>>>>>>>>>>>>>>> [2] http://spark.apache.org/docs/latest/mllib-guide.html >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>> I have been looking into some algorithms related to >>>>>>>>>>>>>>>>> stochastic gradient descent based algorithms.anything i >>>>>>>>>>>>>>>>> should focus please >>>>>>>>>>>>>>>>> let me know.Ans also i will be available for calling this >>>>>>>>>>>>>>>>> week and next >>>>>>>>>>>>>>>>> week.thank you. >>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 5:05 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>> thank you.that's good. i have been trying to fix that for >>>>>>>>>>>>>>>>>> couple of days. please inform me when it will be fixed.now i >>>>>>>>>>>>>>>>>> have been >>>>>>>>>>>>>>>>>> testing the ML algorithms and trying to identify the flow >>>>>>>>>>>>>>>>>> and the >>>>>>>>>>>>>>>>>> hierarchy. is that "Put break points in train methods in >>>>>>>>>>>>>>>>>> Linear Regression >>>>>>>>>>>>>>>>>> class" means the spark/algorithms/ LinearRegrassion.java >>>>>>>>>>>>>>>>>> class in the >>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core? is that the correct file? >>>>>>>>>>>>>>>>>> And also i am planning to write some programs to use >>>>>>>>>>>>>>>>>> apache spark mllib algorithms. and i refer to [1] and some >>>>>>>>>>>>>>>>>> wso2 >>>>>>>>>>>>>>>>>> documentations to get some idea about ML structure.thank you. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1]nirmalfdo.blogspot.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 4:36 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have checked. It seems the issue you have encountered >>>>>>>>>>>>>>>>>>> is cause only in the current development branch of the >>>>>>>>>>>>>>>>>>> product-cep. It >>>>>>>>>>>>>>>>>>> doesn't identify the ML siddhi extension as an extension. >>>>>>>>>>>>>>>>>>> ML siddhi >>>>>>>>>>>>>>>>>>> extension works fine in the latest release of CEP (4.1.0) >>>>>>>>>>>>>>>>>>> [1]. >>>>>>>>>>>>>>>>>>> Until we figure out the reason and come up with a >>>>>>>>>>>>>>>>>>> solution, can you use the latest CEP release for your work. >>>>>>>>>>>>>>>>>>> It's fine to >>>>>>>>>>>>>>>>>>> use that since you haven't started actual development yet. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [1] http://wso2.com/products/complex-event-processor/ >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 3:19 PM, Maheshakya Wijewardena < >>>>>>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Is is vital to use those local repo in my upcoming >>>>>>>>>>>>>>>>>>>>> implementation? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Yes. The remote p2-repo contains the p2-repos of >>>>>>>>>>>>>>>>>>>> released versions. What you have to develop on is the >>>>>>>>>>>>>>>>>>>> current master of the >>>>>>>>>>>>>>>>>>>> carbon-ml and product-ml. You can try out with the >>>>>>>>>>>>>>>>>>>> modification I have >>>>>>>>>>>>>>>>>>>> suggested. In the meantime, I'll verify whether the >>>>>>>>>>>>>>>>>>>> current repos are >>>>>>>>>>>>>>>>>>>> working as expected. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break point in the >>>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> It's great that you have started looking at the >>>>>>>>>>>>>>>>>>>> implementation of linear regression as well. Put break >>>>>>>>>>>>>>>>>>>> points in train >>>>>>>>>>>>>>>>>>>> methods in LinearRegression class. This is being used when >>>>>>>>>>>>>>>>>>>> you run linear >>>>>>>>>>>>>>>>>>>> regression from UI. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I can see some comments left behind for streaming algo >>>>>>>>>>>>>>>>>>>>> as well.thank you >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> You may be referring to the linear regression with SGD >>>>>>>>>>>>>>>>>>>> model. Here, there's no retraining with streaming data >>>>>>>>>>>>>>>>>>>> involved. The SGD >>>>>>>>>>>>>>>>>>>> with minibatches is used to train the model with the data >>>>>>>>>>>>>>>>>>>> set only once. >>>>>>>>>>>>>>>>>>>> What you have to do is create a similar mechanism to >>>>>>>>>>>>>>>>>>>> involve streaming data and retrain models. We will get to >>>>>>>>>>>>>>>>>>>> that part once >>>>>>>>>>>>>>>>>>>> you get comfortable with siddhi extensions. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> BTW, is it possible for you to join a call on this >>>>>>>>>>>>>>>>>>>> Friday or in the next week. We'll try to resolve your >>>>>>>>>>>>>>>>>>>> current issues and >>>>>>>>>>>>>>>>>>>> discuss further on project. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 1:03 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi maheshakya, >>>>>>>>>>>>>>>>>>>>> Is it ok to go with p2 repo at >>>>>>>>>>>>>>>>>>>>> http://product-dist.wso2.com/p2/carbon/releases/wilkes/features/ >>>>>>>>>>>>>>>>>>>>> rather than the P2-repo at >>>>>>>>>>>>>>>>>>>>> product-ml/modules/p2-profile/target/p2-repo in >>>>>>>>>>>>>>>>>>>>> local repo.What is the impact?.Is is vital to use those >>>>>>>>>>>>>>>>>>>>> local repo in my >>>>>>>>>>>>>>>>>>>>> upcoming implementation?so i was trying to give remote p2 >>>>>>>>>>>>>>>>>>>>> repo to cep built >>>>>>>>>>>>>>>>>>>>> by source and trying to debug the cep ml extension and >>>>>>>>>>>>>>>>>>>>> got the same error >>>>>>>>>>>>>>>>>>>>> as yesterday. But pre-built product is working fine. >>>>>>>>>>>>>>>>>>>>> Therefore now i am >>>>>>>>>>>>>>>>>>>>> trying as you described in the last email. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break point in the >>>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression. I am trying to >>>>>>>>>>>>>>>>>>>>> trigger it with >>>>>>>>>>>>>>>>>>>>> product-ml project with data set. Does that Linear >>>>>>>>>>>>>>>>>>>>> Regression also in UI is >>>>>>>>>>>>>>>>>>>>> consuming that spark algorithms or is it in another >>>>>>>>>>>>>>>>>>>>> place? I can see some >>>>>>>>>>>>>>>>>>>>> comments left behind for streaming algo as well.thank you. >>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 9:35 AM, Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>> <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> The earlier error you have mentioned may occur due to >>>>>>>>>>>>>>>>>>>>>> the incompatible Siddhi versions in ML p2-repo and CEP, >>>>>>>>>>>>>>>>>>>>>> when you add the >>>>>>>>>>>>>>>>>>>>>> p2-repo from prodcut-ml you built. >>>>>>>>>>>>>>>>>>>>>> Current siddhi version in product-cep is >>>>>>>>>>>>>>>>>>>>>> 3.0.6-SNAPSHOT[1], but in ML it's 3.0.2. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Can you try changing the siddhi.version in >>>>>>>>>>>>>>>>>>>>>> carbon-ml/pom.xml to 3.0.6-SNAPSHOT, build carbon-ml, >>>>>>>>>>>>>>>>>>>>>> then build product-ml >>>>>>>>>>>>>>>>>>>>>> again. After this, add p2-repo as a local repository >>>>>>>>>>>>>>>>>>>>>> again to fresh CEP >>>>>>>>>>>>>>>>>>>>>> pack and try it out. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mon, May 2, 2016 at 7:02 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>>>> now i remotely debug the CEP extension for ML >>>>>>>>>>>>>>>>>>>>>>> Prediction. What i did was, i have all the pre-build >>>>>>>>>>>>>>>>>>>>>>> version of CEP and ML. >>>>>>>>>>>>>>>>>>>>>>> Therefore i used that built CEP and did the same thing >>>>>>>>>>>>>>>>>>>>>>> that i was doing >>>>>>>>>>>>>>>>>>>>>>> with the source code. I think the only change i did was >>>>>>>>>>>>>>>>>>>>>>> install those >>>>>>>>>>>>>>>>>>>>>>> packages from remote p2 repo. This was work fine and i >>>>>>>>>>>>>>>>>>>>>>> debugged the >>>>>>>>>>>>>>>>>>>>>>> carbon-ml org.wso2.carbon.ml.siddhi.extension as >>>>>>>>>>>>>>>>>>>>>>> described in the [1]. So >>>>>>>>>>>>>>>>>>>>>>> now i have to try same thing with the build by source >>>>>>>>>>>>>>>>>>>>>>> content.thank you. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>> Mahesh >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 5:49 PM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Congratulations and welcome to GSoC 2016. You did a >>>>>>>>>>>>>>>>>>>>>>>> great job in preparing the proposal. Now it's time to >>>>>>>>>>>>>>>>>>>>>>>> dig deep and get >>>>>>>>>>>>>>>>>>>>>>>> started with the project. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> First of all you need to familiarize with the code >>>>>>>>>>>>>>>>>>>>>>>> base. We have agreed to implement this with CEP event >>>>>>>>>>>>>>>>>>>>>>>> streams. We already >>>>>>>>>>>>>>>>>>>>>>>> have a CEP extension for predictions [1][2]. Go >>>>>>>>>>>>>>>>>>>>>>>> through this implementation >>>>>>>>>>>>>>>>>>>>>>>> and familiarize your self with that. You need to >>>>>>>>>>>>>>>>>>>>>>>> understand how: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 1. Even streams are consumed >>>>>>>>>>>>>>>>>>>>>>>> 2. predictions are made from individual event >>>>>>>>>>>>>>>>>>>>>>>> 3. Results are sent back >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Get WSO2 ML and CEP sources (You may use latest >>>>>>>>>>>>>>>>>>>>>>>> released version of CEP) and build the products. Get >>>>>>>>>>>>>>>>>>>>>>>> both carbon-ml[3] and >>>>>>>>>>>>>>>>>>>>>>>> product-ml[4] masters and create new branches for your >>>>>>>>>>>>>>>>>>>>>>>> work from masters. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> After you build the products, you may need to do >>>>>>>>>>>>>>>>>>>>>>>> remote debugging[5] to understand the flow. So please >>>>>>>>>>>>>>>>>>>>>>>> follow an example of >>>>>>>>>>>>>>>>>>>>>>>> real time prediction with ML with debugging and get >>>>>>>>>>>>>>>>>>>>>>>> some idea. The >>>>>>>>>>>>>>>>>>>>>>>> component you need to debug is >>>>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.siddhi.extension. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Next tasks would be implementing online learning >>>>>>>>>>>>>>>>>>>>>>>> algorithms in plain java with spark ml lib and >>>>>>>>>>>>>>>>>>>>>>>> integrating those to ML. We >>>>>>>>>>>>>>>>>>>>>>>> also need to come up with a proper and detailed >>>>>>>>>>>>>>>>>>>>>>>> architecture to employ >>>>>>>>>>>>>>>>>>>>>>>> those algorithms in ML. Getting familiar with the >>>>>>>>>>>>>>>>>>>>>>>> aforementioned sections >>>>>>>>>>>>>>>>>>>>>>>> would give you some insight on how this should be >>>>>>>>>>>>>>>>>>>>>>>> implemented. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> So please try to get a quick grasp then you can >>>>>>>>>>>>>>>>>>>>>>>> start the implementation. Let us know if you have any >>>>>>>>>>>>>>>>>>>>>>>> questions or you get >>>>>>>>>>>>>>>>>>>>>>>> stuck somewhere. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Also, please always add WSO2 developer's list as >>>>>>>>>>>>>>>>>>>>>>>> well when you communicate with us regarding the >>>>>>>>>>>>>>>>>>>>>>>> project so that you can get >>>>>>>>>>>>>>>>>>>>>>>> opinions and feedback from others as well. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>>> https://github.com/wso2/carbon-ml/tree/master/components/extensions/org.wso2.carbon.ml.siddhi.extension >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [3] https://github.com/wso2/carbon-ml >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [4] https://github.com/wso2/product-ml >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [5] >>>>>>>>>>>>>>>>>>>>>>>> https://dzone.com/articles/how-debug-wso2-carbon-kernel >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 3:33 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>> thank you for accepting my GSOC 2016 proposal and >>>>>>>>>>>>>>>>>>>>>>>>> i am looking forward for the further instruction and >>>>>>>>>>>>>>>>>>>>>>>>> project continuation. >>>>>>>>>>>>>>>>>>>>>>>>> thank you very much. >>>>>>>>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>> mahesha...@wso2.com >>>>>>>> +94711228855 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pruthuvi Maheshakya Wijewardena >>>>>> mahesha...@wso2.com >>>>>> +94711228855 >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > > > -- > Pruthuvi Maheshakya Wijewardena > mahesha...@wso2.com > +94711228855 > > > -- Pruthuvi Maheshakya Wijewardena mahesha...@wso2.com +94711228855
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
