Hi Maheshakya, thank you. i will update the repo today.thank you.i changed the carbon ml siddhi extention and see how the changes are effecting. i will update the progress as soon as possible.thank you. i had some problem in spark mllib dependency. i was fixing that. regards, Mahesh. p.s: do i need to maintain a blog?
On Mon, May 16, 2016 at 10:02 AM, Maheshakya Wijewardena < [email protected]> wrote: > Hi Mahesh, > > Sorry for replying late. > > Thank you for the update. I believe you have done some implementations > with with Spark MLLIb algorithms in streaming fashion as we have discussed. > If so, can you please share your code in a Github repo. > > Now i want to implements some machine learning algorithms with importing >> mllib and want to run within your code base >> > > For the moment you can try out editing the same class > PredictStreamProcessor in the siddhi extension in carbon-ml. Later we will > add this separately. You should be able to add org.apache.spark.mllib. > classes to there. > > And i want to see how event streams are coming from cep. As i think it is >> not in a RDD format since it is arriving as the individual samples. I will >> send a email to dev asking about how to get the streams. > > > Please pay attention to length[1] and lengthbatch[1] inbuilt windows in > siddhi. What you need to write are functions similar to a custom aggregate > function[2]. > When you send the email to dev list, explain your requirement. You need to > get a set of event with from a stream with a specified window size (number > of events). Then build a model within that function. You also need to > retain the data (learned weights, cluster centers, etc.) from the previous > window to use in the current window. Ask what can be the most suitable > option for this among the set of siddhi extensions given. > > Best regards. > > [1] > https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength > [2] > https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function > > On Wed, May 11, 2016 at 1:43 PM, Mahesh Dananjaya < > [email protected]> wrote: > >> >> ---------- Forwarded message ---------- >> From: Mahesh Dananjaya <[email protected]> >> Date: Wed, May 11, 2016 at 1:43 PM >> Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive analytic with online >> data for WSO2 Machine Learner >> To: Maheshakya Wijewardena <[email protected]> >> >> >> Hi Maheshakya, >> sorry for not updating. I did what you wanted me to do. I checked the >> code base and train functions. I went through those java docs. I went >> through the carbon-ml current implementation of LG and K-Mean. And i had >> Apache Spark and i tried with several examples. Now i want to implements >> some machine learning algorithms with importing mllib and want to run >> within your code base. Can you help me with that. >> And i want to see how event streams are coming from cep. As i think it is >> not in a RDD format since it is arriving as the individual samples. I will >> send a email to dev asking about how to get the streams. I debugged many of >> those functions in the code base. So need further instructions to >> proceed.thank you. >> regards, >> Mahesh. >> >> On Wed, May 11, 2016 at 10:32 AM, Maheshakya Wijewardena < >> [email protected]> wrote: >> >>> Hi Mahesh, >>> >>> Any update on your progress? >>> >>> Best regards. >>> >>> On Wed, May 4, 2016 at 8:35 PM, Maheshakya Wijewardena < >>> [email protected]> wrote: >>> >>>> Hi Mahesh, >>>> >>>> is that "Put break points in train methods in Linear Regression class" >>>>> means the spark/algorithms/ LinearRegrassion.java class in the >>>>> org.wso2.carbon.ml.core? is that the correct file? >>>> >>>> >>>> Yes, this is the correct place. >>>> >>>> You can refer to spark programming guide[1][2] as well as our ML code >>>> base when you try those algorithms out. Please try to do rough >>>> implementations of the streaming versions of linear regression, logistic >>>> regression and k-means clustering as we have discussed in the proposal in >>>> plain Java. It's better if you can create a git repo and share your code >>>> once you have made some progress. >>>> >>>> Were you able debug and understand the flow of the ML siddhi extension? >>>> I hope you haven't encountered more errors after switching the released >>>> version of CEP. >>>> >>>> Is this Friday okay for you? Afternoon at 2:00 pm? >>>> >>>> Best regards. >>>> >>>> >>>> Best regards. >>>> >>>> [1] http://spark.apache.org/docs/latest/programming-guide.html >>>> [2] http://spark.apache.org/docs/latest/mllib-guide.html >>>> >>>> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >>>> [email protected]> wrote: >>>> >>>>> Hi Maheshakya, >>>>> I have been looking into some algorithms related to stochastic >>>>> gradient descent based algorithms.anything i should focus please let me >>>>> know.Ans also i will be available for calling this week and next >>>>> week.thank >>>>> you. >>>>> BR, >>>>> Mahesh. >>>>> >>>>> On Tue, May 3, 2016 at 5:05 PM, Mahesh Dananjaya < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Maheshakya, >>>>>> thank you.that's good. i have been trying to fix that for couple of >>>>>> days. please inform me when it will be fixed.now i have been testing the >>>>>> ML >>>>>> algorithms and trying to identify the flow and the hierarchy. is that >>>>>> "Put >>>>>> break points in train methods in Linear Regression class" means the >>>>>> spark/algorithms/ LinearRegrassion.java class in the >>>>>> org.wso2.carbon.ml.core? is that the correct file? >>>>>> And also i am planning to write some programs to use apache spark >>>>>> mllib algorithms. and i refer to [1] and some wso2 documentations to get >>>>>> some idea about ML structure.thank you. >>>>>> >>>>>> BR, >>>>>> Mahesh. >>>>>> >>>>>> [1]nirmalfdo.blogspot.com >>>>>> >>>>>> On Tue, May 3, 2016 at 4:36 PM, Maheshakya Wijewardena < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Mahesh, >>>>>>> >>>>>>> I have checked. It seems the issue you have encountered is cause >>>>>>> only in the current development branch of the product-cep. It doesn't >>>>>>> identify the ML siddhi extension as an extension. ML siddhi extension >>>>>>> works >>>>>>> fine in the latest release of CEP (4.1.0) [1]. >>>>>>> Until we figure out the reason and come up with a solution, can you >>>>>>> use the latest CEP release for your work. It's fine to use that since >>>>>>> you >>>>>>> haven't started actual development yet. >>>>>>> >>>>>>> Best regards. >>>>>>> >>>>>>> [1] http://wso2.com/products/complex-event-processor/ >>>>>>> >>>>>>> On Tue, May 3, 2016 at 3:19 PM, Maheshakya Wijewardena < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Mahesh, >>>>>>>> >>>>>>>> >>>>>>>>> Is is vital to use those local repo in my upcoming implementation? >>>>>>>> >>>>>>>> >>>>>>>> Yes. The remote p2-repo contains the p2-repos of released versions. >>>>>>>> What you have to develop on is the current master of the carbon-ml and >>>>>>>> product-ml. You can try out with the modification I have suggested. In >>>>>>>> the >>>>>>>> meantime, I'll verify whether the current repos are working as >>>>>>>> expected. >>>>>>>> >>>>>>>> And also i am trying to debug the carbon-ml org.wso2.carbon.ml.core >>>>>>>>> by putting some break point in the spark/algorithms/Linear Regression >>>>>>>> >>>>>>>> >>>>>>>> It's great that you have started looking at the implementation of >>>>>>>> linear regression as well. Put break points in train methods in >>>>>>>> LinearRegression class. This is being used when you run linear >>>>>>>> regression >>>>>>>> from UI. >>>>>>>> >>>>>>>> I can see some comments left behind for streaming algo as >>>>>>>>> well.thank you >>>>>>>> >>>>>>>> >>>>>>>> You may be referring to the linear regression with SGD model. Here, >>>>>>>> there's no retraining with streaming data involved. The SGD with >>>>>>>> minibatches is used to train the model with the data set only once. >>>>>>>> What you have to do is create a similar mechanism to involve >>>>>>>> streaming data and retrain models. We will get to that part once you >>>>>>>> get >>>>>>>> comfortable with siddhi extensions. >>>>>>>> >>>>>>>> BTW, is it possible for you to join a call on this Friday or in >>>>>>>> the next week. We'll try to resolve your current issues and discuss >>>>>>>> further >>>>>>>> on project. >>>>>>>> >>>>>>>> Best regards. >>>>>>>> >>>>>>>> On Tue, May 3, 2016 at 1:03 PM, Mahesh Dananjaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi maheshakya, >>>>>>>>> Is it ok to go with p2 repo at >>>>>>>>> http://product-dist.wso2.com/p2/carbon/releases/wilkes/features/ >>>>>>>>> rather than the P2-repo at >>>>>>>>> product-ml/modules/p2-profile/target/p2-repo in >>>>>>>>> local repo.What is the impact?.Is is vital to use those local repo in >>>>>>>>> my >>>>>>>>> upcoming implementation?so i was trying to give remote p2 repo to cep >>>>>>>>> built >>>>>>>>> by source and trying to debug the cep ml extension and got the same >>>>>>>>> error >>>>>>>>> as yesterday. But pre-built product is working fine. Therefore now i >>>>>>>>> am >>>>>>>>> trying as you described in the last email. >>>>>>>>> >>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>> org.wso2.carbon.ml.core by putting some break point in the >>>>>>>>> spark/algorithms/Linear Regression. I am trying to trigger it with >>>>>>>>> product-ml project with data set. Does that Linear Regression also in >>>>>>>>> UI is >>>>>>>>> consuming that spark algorithms or is it in another place? I can see >>>>>>>>> some >>>>>>>>> comments left behind for streaming algo as well.thank you. >>>>>>>>> BR, >>>>>>>>> Mahesh. >>>>>>>>> >>>>>>>>> On Tue, May 3, 2016 at 9:35 AM, Maheshakya Wijewardena < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Mahesh, >>>>>>>>>> >>>>>>>>>> The earlier error you have mentioned may occur due to the >>>>>>>>>> incompatible Siddhi versions in ML p2-repo and CEP, when you add the >>>>>>>>>> p2-repo from prodcut-ml you built. >>>>>>>>>> Current siddhi version in product-cep is 3.0.6-SNAPSHOT[1], but >>>>>>>>>> in ML it's 3.0.2. >>>>>>>>>> >>>>>>>>>> Can you try changing the siddhi.version in carbon-ml/pom.xml to >>>>>>>>>> 3.0.6-SNAPSHOT, build carbon-ml, then build product-ml again. After >>>>>>>>>> this, >>>>>>>>>> add p2-repo as a local repository again to fresh CEP pack and try it >>>>>>>>>> out. >>>>>>>>>> >>>>>>>>>> Best regards. >>>>>>>>>> >>>>>>>>>> On Mon, May 2, 2016 at 7:02 PM, Mahesh Dananjaya < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>> now i remotely debug the CEP extension for ML Prediction. What i >>>>>>>>>>> did was, i have all the pre-build version of CEP and ML. Therefore >>>>>>>>>>> i used >>>>>>>>>>> that built CEP and did the same thing that i was doing with the >>>>>>>>>>> source >>>>>>>>>>> code. I think the only change i did was install those packages from >>>>>>>>>>> remote >>>>>>>>>>> p2 repo. This was work fine and i debugged the carbon-ml >>>>>>>>>>> org.wso2.carbon.ml.siddhi.extension as described in the [1]. So now >>>>>>>>>>> i have >>>>>>>>>>> to try same thing with the build by source content.thank you. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>> >>>>>>>>>>> BR, >>>>>>>>>>> Mahesh >>>>>>>>>>> >>>>>>>>>>> On Mon, Apr 25, 2016 at 5:49 PM, Maheshakya Wijewardena < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>> >>>>>>>>>>>> Congratulations and welcome to GSoC 2016. You did a great job >>>>>>>>>>>> in preparing the proposal. Now it's time to dig deep and get >>>>>>>>>>>> started with >>>>>>>>>>>> the project. >>>>>>>>>>>> >>>>>>>>>>>> First of all you need to familiarize with the code base. We >>>>>>>>>>>> have agreed to implement this with CEP event streams. We already >>>>>>>>>>>> have a CEP >>>>>>>>>>>> extension for predictions [1][2]. Go through this implementation >>>>>>>>>>>> and >>>>>>>>>>>> familiarize your self with that. You need to understand how: >>>>>>>>>>>> >>>>>>>>>>>> 1. Even streams are consumed >>>>>>>>>>>> 2. predictions are made from individual event >>>>>>>>>>>> 3. Results are sent back >>>>>>>>>>>> >>>>>>>>>>>> Get WSO2 ML and CEP sources (You may use latest released >>>>>>>>>>>> version of CEP) and build the products. Get both carbon-ml[3] and >>>>>>>>>>>> product-ml[4] masters and create new branches for your work from >>>>>>>>>>>> masters. >>>>>>>>>>>> >>>>>>>>>>>> After you build the products, you may need to do remote >>>>>>>>>>>> debugging[5] to understand the flow. So please follow an example >>>>>>>>>>>> of real >>>>>>>>>>>> time prediction with ML with debugging and get some idea. The >>>>>>>>>>>> component you >>>>>>>>>>>> need to debug is org.wso2.carbon.ml.siddhi.extension. >>>>>>>>>>>> >>>>>>>>>>>> Next tasks would be implementing online learning algorithms in >>>>>>>>>>>> plain java with spark ml lib and integrating those to ML. We also >>>>>>>>>>>> need to >>>>>>>>>>>> come up with a proper and detailed architecture to employ those >>>>>>>>>>>> algorithms >>>>>>>>>>>> in ML. Getting familiar with the aforementioned sections would >>>>>>>>>>>> give you >>>>>>>>>>>> some insight on how this should be implemented. >>>>>>>>>>>> >>>>>>>>>>>> So please try to get a quick grasp then you can start the >>>>>>>>>>>> implementation. Let us know if you have any questions or you get >>>>>>>>>>>> stuck >>>>>>>>>>>> somewhere. >>>>>>>>>>>> >>>>>>>>>>>> Also, please always add WSO2 developer's list as well when you >>>>>>>>>>>> communicate with us regarding the project so that you can get >>>>>>>>>>>> opinions and >>>>>>>>>>>> feedback from others as well. >>>>>>>>>>>> >>>>>>>>>>>> Best regards. >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>> >>>>>>>>>>>> [2] >>>>>>>>>>>> https://github.com/wso2/carbon-ml/tree/master/components/extensions/org.wso2.carbon.ml.siddhi.extension >>>>>>>>>>>> >>>>>>>>>>>> [3] https://github.com/wso2/carbon-ml >>>>>>>>>>>> >>>>>>>>>>>> [4] https://github.com/wso2/product-ml >>>>>>>>>>>> >>>>>>>>>>>> [5] https://dzone.com/articles/how-debug-wso2-carbon-kernel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Apr 25, 2016 at 3:33 PM, Mahesh Dananjaya < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> thank you for accepting my GSOC 2016 proposal and i am looking >>>>>>>>>>>>> forward for the further instruction and project continuation. >>>>>>>>>>>>> thank you >>>>>>>>>>>>> very much. >>>>>>>>>>>>> regards, >>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>> [email protected] >>>>>>>>>>>> +94711228855 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>> [email protected] >>>>>>>>>> +94711228855 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>> [email protected] >>>>>>>> +94711228855 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>> [email protected] >>>>>>> +94711228855 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Pruthuvi Maheshakya Wijewardena >>>> [email protected] >>>> +94711228855 >>>> >>>> >>>> >>> >>> >>> -- >>> Pruthuvi Maheshakya Wijewardena >>> [email protected] >>> +94711228855 >>> >>> >>> >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Pruthuvi Maheshakya Wijewardena > [email protected] > +94711228855 > > >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
