---------- Forwarded message ---------- From: Mahesh Dananjaya <[email protected]> Date: Wed, May 11, 2016 at 1:43 PM Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner To: Maheshakya Wijewardena <[email protected]>
Hi Maheshakya, sorry for not updating. I did what you wanted me to do. I checked the code base and train functions. I went through those java docs. I went through the carbon-ml current implementation of LG and K-Mean. And i had Apache Spark and i tried with several examples. Now i want to implements some machine learning algorithms with importing mllib and want to run within your code base. Can you help me with that. And i want to see how event streams are coming from cep. As i think it is not in a RDD format since it is arriving as the individual samples. I will send a email to dev asking about how to get the streams. I debugged many of those functions in the code base. So need further instructions to proceed.thank you. regards, Mahesh. On Wed, May 11, 2016 at 10:32 AM, Maheshakya Wijewardena < [email protected]> wrote: > Hi Mahesh, > > Any update on your progress? > > Best regards. > > On Wed, May 4, 2016 at 8:35 PM, Maheshakya Wijewardena < > [email protected]> wrote: > >> Hi Mahesh, >> >> is that "Put break points in train methods in Linear Regression class" >>> means the spark/algorithms/ LinearRegrassion.java class in the >>> org.wso2.carbon.ml.core? is that the correct file? >> >> >> Yes, this is the correct place. >> >> You can refer to spark programming guide[1][2] as well as our ML code >> base when you try those algorithms out. Please try to do rough >> implementations of the streaming versions of linear regression, logistic >> regression and k-means clustering as we have discussed in the proposal in >> plain Java. It's better if you can create a git repo and share your code >> once you have made some progress. >> >> Were you able debug and understand the flow of the ML siddhi extension? I >> hope you haven't encountered more errors after switching the released >> version of CEP. >> >> Is this Friday okay for you? Afternoon at 2:00 pm? >> >> Best regards. >> >> >> Best regards. >> >> [1] http://spark.apache.org/docs/latest/programming-guide.html >> [2] http://spark.apache.org/docs/latest/mllib-guide.html >> >> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >> [email protected]> wrote: >> >>> Hi Maheshakya, >>> I have been looking into some algorithms related to stochastic gradient >>> descent based algorithms.anything i should focus please let me know.Ans >>> also i will be available for calling this week and next week.thank you. >>> BR, >>> Mahesh. >>> >>> On Tue, May 3, 2016 at 5:05 PM, Mahesh Dananjaya < >>> [email protected]> wrote: >>> >>>> Hi Maheshakya, >>>> thank you.that's good. i have been trying to fix that for couple of >>>> days. please inform me when it will be fixed.now i have been testing the ML >>>> algorithms and trying to identify the flow and the hierarchy. is that "Put >>>> break points in train methods in Linear Regression class" means the >>>> spark/algorithms/ LinearRegrassion.java class in the >>>> org.wso2.carbon.ml.core? is that the correct file? >>>> And also i am planning to write some programs to use apache spark mllib >>>> algorithms. and i refer to [1] and some wso2 documentations to get some >>>> idea about ML structure.thank you. >>>> >>>> BR, >>>> Mahesh. >>>> >>>> [1]nirmalfdo.blogspot.com >>>> >>>> On Tue, May 3, 2016 at 4:36 PM, Maheshakya Wijewardena < >>>> [email protected]> wrote: >>>> >>>>> Hi Mahesh, >>>>> >>>>> I have checked. It seems the issue you have encountered is cause only >>>>> in the current development branch of the product-cep. It doesn't identify >>>>> the ML siddhi extension as an extension. ML siddhi extension works fine in >>>>> the latest release of CEP (4.1.0) [1]. >>>>> Until we figure out the reason and come up with a solution, can you >>>>> use the latest CEP release for your work. It's fine to use that since you >>>>> haven't started actual development yet. >>>>> >>>>> Best regards. >>>>> >>>>> [1] http://wso2.com/products/complex-event-processor/ >>>>> >>>>> On Tue, May 3, 2016 at 3:19 PM, Maheshakya Wijewardena < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Mahesh, >>>>>> >>>>>> >>>>>>> Is is vital to use those local repo in my upcoming implementation? >>>>>> >>>>>> >>>>>> Yes. The remote p2-repo contains the p2-repos of released versions. >>>>>> What you have to develop on is the current master of the carbon-ml and >>>>>> product-ml. You can try out with the modification I have suggested. In >>>>>> the >>>>>> meantime, I'll verify whether the current repos are working as expected. >>>>>> >>>>>> And also i am trying to debug the carbon-ml org.wso2.carbon.ml.core >>>>>>> by putting some break point in the spark/algorithms/Linear Regression >>>>>> >>>>>> >>>>>> It's great that you have started looking at the implementation of >>>>>> linear regression as well. Put break points in train methods in >>>>>> LinearRegression class. This is being used when you run linear regression >>>>>> from UI. >>>>>> >>>>>> I can see some comments left behind for streaming algo as well.thank >>>>>>> you >>>>>> >>>>>> >>>>>> You may be referring to the linear regression with SGD model. Here, >>>>>> there's no retraining with streaming data involved. The SGD with >>>>>> minibatches is used to train the model with the data set only once. >>>>>> What you have to do is create a similar mechanism to involve >>>>>> streaming data and retrain models. We will get to that part once you get >>>>>> comfortable with siddhi extensions. >>>>>> >>>>>> BTW, is it possible for you to join a call on this Friday or in the >>>>>> next week. We'll try to resolve your current issues and discuss further >>>>>> on >>>>>> project. >>>>>> >>>>>> Best regards. >>>>>> >>>>>> On Tue, May 3, 2016 at 1:03 PM, Mahesh Dananjaya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi maheshakya, >>>>>>> Is it ok to go with p2 repo at >>>>>>> http://product-dist.wso2.com/p2/carbon/releases/wilkes/features/ >>>>>>> rather than the P2-repo at product-ml/modules/p2-profile/target/p2-repo >>>>>>> in >>>>>>> local repo.What is the impact?.Is is vital to use those local repo in my >>>>>>> upcoming implementation?so i was trying to give remote p2 repo to cep >>>>>>> built >>>>>>> by source and trying to debug the cep ml extension and got the same >>>>>>> error >>>>>>> as yesterday. But pre-built product is working fine. Therefore now i am >>>>>>> trying as you described in the last email. >>>>>>> >>>>>>> And also i am trying to debug the carbon-ml org.wso2.carbon.ml.core >>>>>>> by putting some break point in the spark/algorithms/Linear Regression. >>>>>>> I am >>>>>>> trying to trigger it with product-ml project with data set. Does that >>>>>>> Linear Regression also in UI is consuming that spark algorithms or is >>>>>>> it in >>>>>>> another place? I can see some comments left behind for streaming algo as >>>>>>> well.thank you. >>>>>>> BR, >>>>>>> Mahesh. >>>>>>> >>>>>>> On Tue, May 3, 2016 at 9:35 AM, Maheshakya Wijewardena < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Mahesh, >>>>>>>> >>>>>>>> The earlier error you have mentioned may occur due to the >>>>>>>> incompatible Siddhi versions in ML p2-repo and CEP, when you add the >>>>>>>> p2-repo from prodcut-ml you built. >>>>>>>> Current siddhi version in product-cep is 3.0.6-SNAPSHOT[1], but in >>>>>>>> ML it's 3.0.2. >>>>>>>> >>>>>>>> Can you try changing the siddhi.version in carbon-ml/pom.xml to >>>>>>>> 3.0.6-SNAPSHOT, build carbon-ml, then build product-ml again. After >>>>>>>> this, >>>>>>>> add p2-repo as a local repository again to fresh CEP pack and try it >>>>>>>> out. >>>>>>>> >>>>>>>> Best regards. >>>>>>>> >>>>>>>> On Mon, May 2, 2016 at 7:02 PM, Mahesh Dananjaya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Maheshakya, >>>>>>>>> now i remotely debug the CEP extension for ML Prediction. What i >>>>>>>>> did was, i have all the pre-build version of CEP and ML. Therefore i >>>>>>>>> used >>>>>>>>> that built CEP and did the same thing that i was doing with the source >>>>>>>>> code. I think the only change i did was install those packages from >>>>>>>>> remote >>>>>>>>> p2 repo. This was work fine and i debugged the carbon-ml >>>>>>>>> org.wso2.carbon.ml.siddhi.extension as described in the [1]. So now i >>>>>>>>> have >>>>>>>>> to try same thing with the build by source content.thank you. >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>> >>>>>>>>> BR, >>>>>>>>> Mahesh >>>>>>>>> >>>>>>>>> On Mon, Apr 25, 2016 at 5:49 PM, Maheshakya Wijewardena < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Mahesh, >>>>>>>>>> >>>>>>>>>> Congratulations and welcome to GSoC 2016. You did a great job in >>>>>>>>>> preparing the proposal. Now it's time to dig deep and get started >>>>>>>>>> with the >>>>>>>>>> project. >>>>>>>>>> >>>>>>>>>> First of all you need to familiarize with the code base. We have >>>>>>>>>> agreed to implement this with CEP event streams. We already have a >>>>>>>>>> CEP >>>>>>>>>> extension for predictions [1][2]. Go through this implementation and >>>>>>>>>> familiarize your self with that. You need to understand how: >>>>>>>>>> >>>>>>>>>> 1. Even streams are consumed >>>>>>>>>> 2. predictions are made from individual event >>>>>>>>>> 3. Results are sent back >>>>>>>>>> >>>>>>>>>> Get WSO2 ML and CEP sources (You may use latest released version >>>>>>>>>> of CEP) and build the products. Get both carbon-ml[3] and >>>>>>>>>> product-ml[4] >>>>>>>>>> masters and create new branches for your work from masters. >>>>>>>>>> >>>>>>>>>> After you build the products, you may need to do remote >>>>>>>>>> debugging[5] to understand the flow. So please follow an example of >>>>>>>>>> real >>>>>>>>>> time prediction with ML with debugging and get some idea. The >>>>>>>>>> component you >>>>>>>>>> need to debug is org.wso2.carbon.ml.siddhi.extension. >>>>>>>>>> >>>>>>>>>> Next tasks would be implementing online learning algorithms in >>>>>>>>>> plain java with spark ml lib and integrating those to ML. We also >>>>>>>>>> need to >>>>>>>>>> come up with a proper and detailed architecture to employ those >>>>>>>>>> algorithms >>>>>>>>>> in ML. Getting familiar with the aforementioned sections would give >>>>>>>>>> you >>>>>>>>>> some insight on how this should be implemented. >>>>>>>>>> >>>>>>>>>> So please try to get a quick grasp then you can start the >>>>>>>>>> implementation. Let us know if you have any questions or you get >>>>>>>>>> stuck >>>>>>>>>> somewhere. >>>>>>>>>> >>>>>>>>>> Also, please always add WSO2 developer's list as well when you >>>>>>>>>> communicate with us regarding the project so that you can get >>>>>>>>>> opinions and >>>>>>>>>> feedback from others as well. >>>>>>>>>> >>>>>>>>>> Best regards. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>> >>>>>>>>>> [2] >>>>>>>>>> https://github.com/wso2/carbon-ml/tree/master/components/extensions/org.wso2.carbon.ml.siddhi.extension >>>>>>>>>> >>>>>>>>>> [3] https://github.com/wso2/carbon-ml >>>>>>>>>> >>>>>>>>>> [4] https://github.com/wso2/product-ml >>>>>>>>>> >>>>>>>>>> [5] https://dzone.com/articles/how-debug-wso2-carbon-kernel >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Apr 25, 2016 at 3:33 PM, Mahesh Dananjaya < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> thank you for accepting my GSOC 2016 proposal and i am looking >>>>>>>>>>> forward for the further instruction and project continuation. thank >>>>>>>>>>> you >>>>>>>>>>> very much. >>>>>>>>>>> regards, >>>>>>>>>>> Mahesh. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>> [email protected] >>>>>>>>>> +94711228855 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>> [email protected] >>>>>>>> +94711228855 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pruthuvi Maheshakya Wijewardena >>>>>> [email protected] >>>>>> +94711228855 >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pruthuvi Maheshakya Wijewardena >>>>> [email protected] >>>>> +94711228855 >>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> Pruthuvi Maheshakya Wijewardena >> [email protected] >> +94711228855 >> >> >> > > > -- > Pruthuvi Maheshakya Wijewardena > [email protected] > +94711228855 > > >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
