Hi Mahesh, Sorry for replying late.
Thank you for the update. I believe you have done some implementations with with Spark MLLIb algorithms in streaming fashion as we have discussed. If so, can you please share your code in a Github repo. Now i want to implements some machine learning algorithms with importing > mllib and want to run within your code base > For the moment you can try out editing the same class PredictStreamProcessor in the siddhi extension in carbon-ml. Later we will add this separately. You should be able to add org.apache.spark.mllib. classes to there. And i want to see how event streams are coming from cep. As i think it is > not in a RDD format since it is arriving as the individual samples. I will > send a email to dev asking about how to get the streams. Please pay attention to length[1] and lengthbatch[1] inbuilt windows in siddhi. What you need to write are functions similar to a custom aggregate function[2]. When you send the email to dev list, explain your requirement. You need to get a set of event with from a stream with a specified window size (number of events). Then build a model within that function. You also need to retain the data (learned weights, cluster centers, etc.) from the previous window to use in the current window. Ask what can be the most suitable option for this among the set of siddhi extensions given. Best regards. [1] https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength [2] https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function On Wed, May 11, 2016 at 1:43 PM, Mahesh Dananjaya <[email protected] > wrote: > > ---------- Forwarded message ---------- > From: Mahesh Dananjaya <[email protected]> > Date: Wed, May 11, 2016 at 1:43 PM > Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive analytic with online > data for WSO2 Machine Learner > To: Maheshakya Wijewardena <[email protected]> > > > Hi Maheshakya, > sorry for not updating. I did what you wanted me to do. I checked the code > base and train functions. I went through those java docs. I went through > the carbon-ml current implementation of LG and K-Mean. And i had Apache > Spark and i tried with several examples. Now i want to implements some > machine learning algorithms with importing mllib and want to run within > your code base. Can you help me with that. > And i want to see how event streams are coming from cep. As i think it is > not in a RDD format since it is arriving as the individual samples. I will > send a email to dev asking about how to get the streams. I debugged many of > those functions in the code base. So need further instructions to > proceed.thank you. > regards, > Mahesh. > > On Wed, May 11, 2016 at 10:32 AM, Maheshakya Wijewardena < > [email protected]> wrote: > >> Hi Mahesh, >> >> Any update on your progress? >> >> Best regards. >> >> On Wed, May 4, 2016 at 8:35 PM, Maheshakya Wijewardena < >> [email protected]> wrote: >> >>> Hi Mahesh, >>> >>> is that "Put break points in train methods in Linear Regression class" >>>> means the spark/algorithms/ LinearRegrassion.java class in the >>>> org.wso2.carbon.ml.core? is that the correct file? >>> >>> >>> Yes, this is the correct place. >>> >>> You can refer to spark programming guide[1][2] as well as our ML code >>> base when you try those algorithms out. Please try to do rough >>> implementations of the streaming versions of linear regression, logistic >>> regression and k-means clustering as we have discussed in the proposal in >>> plain Java. It's better if you can create a git repo and share your code >>> once you have made some progress. >>> >>> Were you able debug and understand the flow of the ML siddhi extension? >>> I hope you haven't encountered more errors after switching the released >>> version of CEP. >>> >>> Is this Friday okay for you? Afternoon at 2:00 pm? >>> >>> Best regards. >>> >>> >>> Best regards. >>> >>> [1] http://spark.apache.org/docs/latest/programming-guide.html >>> [2] http://spark.apache.org/docs/latest/mllib-guide.html >>> >>> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >>> [email protected]> wrote: >>> >>>> Hi Maheshakya, >>>> I have been looking into some algorithms related to stochastic gradient >>>> descent based algorithms.anything i should focus please let me know.Ans >>>> also i will be available for calling this week and next week.thank you. >>>> BR, >>>> Mahesh. >>>> >>>> On Tue, May 3, 2016 at 5:05 PM, Mahesh Dananjaya < >>>> [email protected]> wrote: >>>> >>>>> Hi Maheshakya, >>>>> thank you.that's good. i have been trying to fix that for couple of >>>>> days. please inform me when it will be fixed.now i have been testing the >>>>> ML >>>>> algorithms and trying to identify the flow and the hierarchy. is that "Put >>>>> break points in train methods in Linear Regression class" means the >>>>> spark/algorithms/ LinearRegrassion.java class in the >>>>> org.wso2.carbon.ml.core? is that the correct file? >>>>> And also i am planning to write some programs to use apache spark >>>>> mllib algorithms. and i refer to [1] and some wso2 documentations to get >>>>> some idea about ML structure.thank you. >>>>> >>>>> BR, >>>>> Mahesh. >>>>> >>>>> [1]nirmalfdo.blogspot.com >>>>> >>>>> On Tue, May 3, 2016 at 4:36 PM, Maheshakya Wijewardena < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Mahesh, >>>>>> >>>>>> I have checked. It seems the issue you have encountered is cause only >>>>>> in the current development branch of the product-cep. It doesn't identify >>>>>> the ML siddhi extension as an extension. ML siddhi extension works fine >>>>>> in >>>>>> the latest release of CEP (4.1.0) [1]. >>>>>> Until we figure out the reason and come up with a solution, can you >>>>>> use the latest CEP release for your work. It's fine to use that since you >>>>>> haven't started actual development yet. >>>>>> >>>>>> Best regards. >>>>>> >>>>>> [1] http://wso2.com/products/complex-event-processor/ >>>>>> >>>>>> On Tue, May 3, 2016 at 3:19 PM, Maheshakya Wijewardena < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Mahesh, >>>>>>> >>>>>>> >>>>>>>> Is is vital to use those local repo in my upcoming implementation? >>>>>>> >>>>>>> >>>>>>> Yes. The remote p2-repo contains the p2-repos of released versions. >>>>>>> What you have to develop on is the current master of the carbon-ml and >>>>>>> product-ml. You can try out with the modification I have suggested. In >>>>>>> the >>>>>>> meantime, I'll verify whether the current repos are working as expected. >>>>>>> >>>>>>> And also i am trying to debug the carbon-ml org.wso2.carbon.ml.core >>>>>>>> by putting some break point in the spark/algorithms/Linear Regression >>>>>>> >>>>>>> >>>>>>> It's great that you have started looking at the implementation of >>>>>>> linear regression as well. Put break points in train methods in >>>>>>> LinearRegression class. This is being used when you run linear >>>>>>> regression >>>>>>> from UI. >>>>>>> >>>>>>> I can see some comments left behind for streaming algo as well.thank >>>>>>>> you >>>>>>> >>>>>>> >>>>>>> You may be referring to the linear regression with SGD model. Here, >>>>>>> there's no retraining with streaming data involved. The SGD with >>>>>>> minibatches is used to train the model with the data set only once. >>>>>>> What you have to do is create a similar mechanism to involve >>>>>>> streaming data and retrain models. We will get to that part once you get >>>>>>> comfortable with siddhi extensions. >>>>>>> >>>>>>> BTW, is it possible for you to join a call on this Friday or in the >>>>>>> next week. We'll try to resolve your current issues and discuss further >>>>>>> on >>>>>>> project. >>>>>>> >>>>>>> Best regards. >>>>>>> >>>>>>> On Tue, May 3, 2016 at 1:03 PM, Mahesh Dananjaya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi maheshakya, >>>>>>>> Is it ok to go with p2 repo at >>>>>>>> http://product-dist.wso2.com/p2/carbon/releases/wilkes/features/ >>>>>>>> rather than the P2-repo at >>>>>>>> product-ml/modules/p2-profile/target/p2-repo in >>>>>>>> local repo.What is the impact?.Is is vital to use those local repo in >>>>>>>> my >>>>>>>> upcoming implementation?so i was trying to give remote p2 repo to cep >>>>>>>> built >>>>>>>> by source and trying to debug the cep ml extension and got the same >>>>>>>> error >>>>>>>> as yesterday. But pre-built product is working fine. Therefore now i >>>>>>>> am >>>>>>>> trying as you described in the last email. >>>>>>>> >>>>>>>> And also i am trying to debug the carbon-ml org.wso2.carbon.ml.core >>>>>>>> by putting some break point in the spark/algorithms/Linear Regression. >>>>>>>> I am >>>>>>>> trying to trigger it with product-ml project with data set. Does that >>>>>>>> Linear Regression also in UI is consuming that spark algorithms or is >>>>>>>> it in >>>>>>>> another place? I can see some comments left behind for streaming algo >>>>>>>> as >>>>>>>> well.thank you. >>>>>>>> BR, >>>>>>>> Mahesh. >>>>>>>> >>>>>>>> On Tue, May 3, 2016 at 9:35 AM, Maheshakya Wijewardena < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Mahesh, >>>>>>>>> >>>>>>>>> The earlier error you have mentioned may occur due to the >>>>>>>>> incompatible Siddhi versions in ML p2-repo and CEP, when you add the >>>>>>>>> p2-repo from prodcut-ml you built. >>>>>>>>> Current siddhi version in product-cep is 3.0.6-SNAPSHOT[1], but in >>>>>>>>> ML it's 3.0.2. >>>>>>>>> >>>>>>>>> Can you try changing the siddhi.version in carbon-ml/pom.xml to >>>>>>>>> 3.0.6-SNAPSHOT, build carbon-ml, then build product-ml again. After >>>>>>>>> this, >>>>>>>>> add p2-repo as a local repository again to fresh CEP pack and try it >>>>>>>>> out. >>>>>>>>> >>>>>>>>> Best regards. >>>>>>>>> >>>>>>>>> On Mon, May 2, 2016 at 7:02 PM, Mahesh Dananjaya < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Maheshakya, >>>>>>>>>> now i remotely debug the CEP extension for ML Prediction. What i >>>>>>>>>> did was, i have all the pre-build version of CEP and ML. Therefore i >>>>>>>>>> used >>>>>>>>>> that built CEP and did the same thing that i was doing with the >>>>>>>>>> source >>>>>>>>>> code. I think the only change i did was install those packages from >>>>>>>>>> remote >>>>>>>>>> p2 repo. This was work fine and i debugged the carbon-ml >>>>>>>>>> org.wso2.carbon.ml.siddhi.extension as described in the [1]. So now >>>>>>>>>> i have >>>>>>>>>> to try same thing with the build by source content.thank you. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>> >>>>>>>>>> BR, >>>>>>>>>> Mahesh >>>>>>>>>> >>>>>>>>>> On Mon, Apr 25, 2016 at 5:49 PM, Maheshakya Wijewardena < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Mahesh, >>>>>>>>>>> >>>>>>>>>>> Congratulations and welcome to GSoC 2016. You did a great job in >>>>>>>>>>> preparing the proposal. Now it's time to dig deep and get started >>>>>>>>>>> with the >>>>>>>>>>> project. >>>>>>>>>>> >>>>>>>>>>> First of all you need to familiarize with the code base. We have >>>>>>>>>>> agreed to implement this with CEP event streams. We already have a >>>>>>>>>>> CEP >>>>>>>>>>> extension for predictions [1][2]. Go through this implementation and >>>>>>>>>>> familiarize your self with that. You need to understand how: >>>>>>>>>>> >>>>>>>>>>> 1. Even streams are consumed >>>>>>>>>>> 2. predictions are made from individual event >>>>>>>>>>> 3. Results are sent back >>>>>>>>>>> >>>>>>>>>>> Get WSO2 ML and CEP sources (You may use latest released version >>>>>>>>>>> of CEP) and build the products. Get both carbon-ml[3] and >>>>>>>>>>> product-ml[4] >>>>>>>>>>> masters and create new branches for your work from masters. >>>>>>>>>>> >>>>>>>>>>> After you build the products, you may need to do remote >>>>>>>>>>> debugging[5] to understand the flow. So please follow an example of >>>>>>>>>>> real >>>>>>>>>>> time prediction with ML with debugging and get some idea. The >>>>>>>>>>> component you >>>>>>>>>>> need to debug is org.wso2.carbon.ml.siddhi.extension. >>>>>>>>>>> >>>>>>>>>>> Next tasks would be implementing online learning algorithms in >>>>>>>>>>> plain java with spark ml lib and integrating those to ML. We also >>>>>>>>>>> need to >>>>>>>>>>> come up with a proper and detailed architecture to employ those >>>>>>>>>>> algorithms >>>>>>>>>>> in ML. Getting familiar with the aforementioned sections would give >>>>>>>>>>> you >>>>>>>>>>> some insight on how this should be implemented. >>>>>>>>>>> >>>>>>>>>>> So please try to get a quick grasp then you can start the >>>>>>>>>>> implementation. Let us know if you have any questions or you get >>>>>>>>>>> stuck >>>>>>>>>>> somewhere. >>>>>>>>>>> >>>>>>>>>>> Also, please always add WSO2 developer's list as well when you >>>>>>>>>>> communicate with us regarding the project so that you can get >>>>>>>>>>> opinions and >>>>>>>>>>> feedback from others as well. >>>>>>>>>>> >>>>>>>>>>> Best regards. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>> >>>>>>>>>>> [2] >>>>>>>>>>> https://github.com/wso2/carbon-ml/tree/master/components/extensions/org.wso2.carbon.ml.siddhi.extension >>>>>>>>>>> >>>>>>>>>>> [3] https://github.com/wso2/carbon-ml >>>>>>>>>>> >>>>>>>>>>> [4] https://github.com/wso2/product-ml >>>>>>>>>>> >>>>>>>>>>> [5] https://dzone.com/articles/how-debug-wso2-carbon-kernel >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Apr 25, 2016 at 3:33 PM, Mahesh Dananjaya < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> thank you for accepting my GSOC 2016 proposal and i am looking >>>>>>>>>>>> forward for the further instruction and project continuation. >>>>>>>>>>>> thank you >>>>>>>>>>>> very much. >>>>>>>>>>>> regards, >>>>>>>>>>>> Mahesh. >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>> [email protected] >>>>>>>>>>> +94711228855 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>> [email protected] >>>>>>>>> +94711228855 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>> [email protected] >>>>>>> +94711228855 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Pruthuvi Maheshakya Wijewardena >>>>>> [email protected] >>>>>> +94711228855 >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Pruthuvi Maheshakya Wijewardena >>> [email protected] >>> +94711228855 >>> >>> >>> >> >> >> -- >> Pruthuvi Maheshakya Wijewardena >> [email protected] >> +94711228855 >> >> >> > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Pruthuvi Maheshakya Wijewardena [email protected] +94711228855
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
