Hi Maheshakya, I have looked into the spark streaming fundamentals and k mean clustering to develop the streaming k mean clustering for stream data. those can be found at [1] and [2].I will commit new changes to my repo today including the basic implementation of streaming k mean clustering.thank you. regards, Mahesh. [1] http://spark.apache.org/docs/latest/streaming-programming-guide.html [2] http://spark.apache.org/docs/latest/mllib-clustering.html
On Sat, Jun 4, 2016 at 10:51 AM, Mahesh Dananjaya <dananjayamah...@gmail.com > wrote: > Hi Maheshakya, > As you requested i can change other parameters as well such as feature > size(p). Initially i did it with p=3;sure thing. Anyway you can see and run > the code if you want. source is at [1]. the test timing is called with > random data as you requested if you set args[0] to 1. And you can find the > extension and streaming algorithms in gsoc/ directiry[2]. thank you. > BR, > Mahesh. > [1] > https://github.com/dananjayamahesh/GSOC2016/blob/master/spark-examples/first-example/src/main/java/org/sparkexample/StreamingLinearRegression.java > [2] https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc > > On Sat, Jun 4, 2016 at 10:39 AM, Mahesh Dananjaya < > dananjayamah...@gmail.com> wrote: > >> Hi supun, >> Though i pushed it yesterday, there was some problems with the network. >> now you can see them in the repo location [1].I added some Matlab plot you >> can see the patter there.you can use ml also. Ok sure thing. I can prepare >> a report or else blog if you want. files are as follows. The y axis is in >> ns and x axis is in batch size. And also i added two pplots as jpegs[2], so >> you can easily compare. >> lr_timing_1000.txt -> batch size incremented by 1000 >> lr_timing_10000.txt -> batch size incremented by 10000 >> lr_timing_power10.txt -> batch size incremented by power of 10 >> >> In here independent variable is only tha batch size.If you want i can >> send you making other parameters such as step size, number of iteration, >> feature vector size as independent variables. please let me know if you >> want further info. thank you. >> regards, >> Mahesh. >> >> >> [1 >> ]https://github.com/dananjayamahesh/GSOC2016/tree/master/spark-examples/first-example/output >> [2] >> https://github.com/dananjayamahesh/GSOC2016/blob/master/spark-examples/first-example/output/lr_timing_10000.jpg >> >> On Sat, Jun 4, 2016 at 9:58 AM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Mahesh, >>> >>> I have added those timing reports to my repo [1]. >>> >>> Whats the file name? :) >>> >>> Btw, can you compile simple doc (gdoc) with the above results, and bring >>> everything to one place? That way it is easy to compare, and keep track. >>> >>> Thanks, >>> Supun >>> >>> On Fri, Jun 3, 2016 at 7:23 PM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi Maheshkya, >>>> I have added those timing reports to my repo [1].please have a look at. >>>> three files are there. one is using incremet as 1000 for batch sizes >>>> (lr_timing_1000). Otherone is using incremet by 10000 (lr_timing_10000) >>>> upto 1 million in both scenarios.you can see the reports and figures in the >>>> location [2] in the repo. i also added the streaminglinearregression >>>> classes in the repo gsoc folder.thank you. >>>> regards, >>>> Mahesh. >>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>> [2] >>>> https://github.com/dananjayamahesh/GSOC2016/tree/master/spark-examples/first-example/output >>>> >>>> On Mon, May 30, 2016 at 9:24 AM, Maheshakya Wijewardena < >>>> mahesha...@wso2.com> wrote: >>>> >>>>> Hi Mahesh, >>>>> >>>>> Thank you for the update. I will look into your implementation. >>>>> >>>>> And i will be able to send you the timing/performances analysis report >>>>>> tomorrow for the SGD functions >>>>>> >>>>> >>>>> Great. Sent those asap so that we can proceed. >>>>> >>>>> Best regards. >>>>> >>>>> On Sun, May 29, 2016 at 6:56 PM, Mahesh Dananjaya < >>>>> dananjayamah...@gmail.com> wrote: >>>>> >>>>>> >>>>>> Hi maheshakay, >>>>>> I have implemented the linear regression with cep siddhi event stream >>>>>> with taking batch sizes as parameters from the cep. Now we can trying >>>>>> the >>>>>> moving window method to. Before that i think i should get your opinion on >>>>>> data structures to save the streaming data.please check my repo [1] >>>>>> /gsoc/ >>>>>> folder there you can find all new things i add.. there in the extension >>>>>> folder you can find those extension. And i will be able to send you the >>>>>> timing/performances analysis report tomorrow for the SGD functions. thank >>>>>> you. >>>>>> regards, >>>>>> Mahesh. >>>>>> [1] https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc >>>>>> >>>>>> >>>>>> On Fri, May 27, 2016 at 12:56 PM, Mahesh Dananjaya < >>>>>> dananjayamah...@gmail.com> wrote: >>>>>> >>>>>>> Hi maheshkaya, >>>>>>> i have written some siddhi extension and trying to develop a one for >>>>>>> my one. In time series example in the [1], can you please explain me the >>>>>>> input format and query lines in that example for my understanding. >>>>>>> >>>>>>> from baseballData#timeseries:regress(2, 10000, 0.95, salary, rbi, >>>>>>> walks, strikeouts, errors) >>>>>>> select * >>>>>>> insert into regResults; >>>>>>> >>>>>>> i just want to knwo how i give a set of data into this extension and >>>>>>> what is baseballData. Is it input stream as usual.or any data file?how >>>>>>> can >>>>>>> i find that data set to create dummy input stream like baseballData? >>>>>>> >>>>>>> thank you. >>>>>>> regards, >>>>>>> Mahesh. >>>>>>> [1] >>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >>>>>>> >>>>>>> On Thu, May 26, 2016 at 2:58 PM, Mahesh Dananjaya < >>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Maheshakya, >>>>>>>> today i got the siddhi and debug the math extention. then did some >>>>>>>> changes and check. Now i am trying to write same kind of extension in >>>>>>>> my >>>>>>>> code base. so i add dependencies and it was built fine. Now i am >>>>>>>> trying to >>>>>>>> debug my extension and i did the same thing as i did in previous case. >>>>>>>> Cep >>>>>>>> is sending data, bu my extension is not firing in relevant break point. >>>>>>>> 1. So how can i debug the siddhi extension in my new extension.(you >>>>>>>> can see it in my example repoo) >>>>>>>> >>>>>>>> I think if i do it correctly we can built the extension for our >>>>>>>> purpose. And i will send the relevant timing report of SGD algorithms >>>>>>>> very >>>>>>>> soon as supun was asking me. thank you. >>>>>>>> regards, >>>>>>>> Mahesh. >>>>>>>> >>>>>>>> On Tue, May 24, 2016 at 11:07 AM, Maheshakya Wijewardena < >>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>> >>>>>>>>> Also note that there is a calculation interval in the siddhi time >>>>>>>>> series regression function[1]. You maybe able get some insight for >>>>>>>>> this >>>>>>>>> from that as well. >>>>>>>>> >>>>>>>>> [1] https://docs.wso2.com/display/CEP400/Regression >>>>>>>>> >>>>>>>>> On Tue, May 24, 2016 at 11:03 AM, Maheshakya Wijewardena < >>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Mahesh, >>>>>>>>>> >>>>>>>>>> As we discussed offline, we can use similar mechanism to train >>>>>>>>>> linear regression models, logistic regression models and k-means >>>>>>>>>> clustering >>>>>>>>>> models. >>>>>>>>>> >>>>>>>>>> It is very interesting that i have found that somethings that can >>>>>>>>>>> make use of our work. In the cep 4.0 documentation there is a >>>>>>>>>>> Custom Stream >>>>>>>>>>> Processor Extention program [1]. There is a example of >>>>>>>>>>> LinearRegressionStreamProcessor [1]. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> As we have to train predictive models with Spark, you can write >>>>>>>>>> wrappers around regression/clustering models of Spark. Refer to >>>>>>>>>> Siddhi time >>>>>>>>>> series regression source codes[1][2]. You can write a streaming >>>>>>>>>> linear >>>>>>>>>> regression class for ML in a similar fashion by wrapping Spark mllib >>>>>>>>>> implementations. You can use the methods "addEvent", "removeEvent", >>>>>>>>>> etc. >>>>>>>>>> (may have to be changed according to requirements) for the similar >>>>>>>>>> purpose. >>>>>>>>>> You can introduce trainLinearRegression/LogisticRegression/Kmeans >>>>>>>>>> which >>>>>>>>>> does a similar thing as in createLinearRegression in those time >>>>>>>>>> series >>>>>>>>>> functions. In the processData method you can use Spark mllib classes >>>>>>>>>> to >>>>>>>>>> actually train models and return the model weights, evaluation >>>>>>>>>> metrics. So, >>>>>>>>>> converting streams into RDDs and retrieving information from the >>>>>>>>>> trained >>>>>>>>>> models shall happen in this method. >>>>>>>>>> >>>>>>>>>> In the stream processor extension example, you can retrieve those >>>>>>>>>> values then use them to train new models with new batches. >>>>>>>>>> Weights/cluster >>>>>>>>>> centers maybe passed as initialization parameters for the wrappers. >>>>>>>>>> >>>>>>>>>> Please note that we have to figure out the best siddhi extension >>>>>>>>>> type for this process. In the siddhi query, we define batch size, >>>>>>>>>> type of >>>>>>>>>> algorithm and number of features (there can be more). After batch >>>>>>>>>> size >>>>>>>>>> number of events received, train a model and save parameters, return >>>>>>>>>> evaluation metric. With the next batch, retrain the model >>>>>>>>>> initialized with >>>>>>>>>> previously learned parameters. >>>>>>>>>> >>>>>>>>>> We also may need to test the same scenario with a moving window, >>>>>>>>>> but I suspect that that approach may become so slow as a model is >>>>>>>>>> trained >>>>>>>>>> each time an event is received. So, we may have to change the number >>>>>>>>>> of >>>>>>>>>> slots the moving window moves at a time (eg: not one by one, but ten >>>>>>>>>> by >>>>>>>>>> ten). >>>>>>>>>> >>>>>>>>>> Once this is resolved, majority of the research part will be >>>>>>>>>> finished and all we will be left to do is implementing wrappers >>>>>>>>>> around the >>>>>>>>>> 3 learning algorithms we consider. >>>>>>>>>> >>>>>>>>>> Best regards. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/RegressionCalculator.java >>>>>>>>>> [2] >>>>>>>>>> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/timeseries/src/main/java/org/wso2/siddhi/extension/timeseries/linreg/SimpleLinearRegressionCalculator.java >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, May 21, 2016 at 2:55 PM, Mahesh Dananjaya < >>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Maheshkya, >>>>>>>>>>> shall we use [1] for our work? i am checking the possibility. >>>>>>>>>>> BR, >>>>>>>>>>> Mahesh. >>>>>>>>>>> [1] >>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >>>>>>>>>>> [2] >>>>>>>>>>> https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength >>>>>>>>>>> [3] >>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function >>>>>>>>>>> >>>>>>>>>>> On Sat, May 21, 2016 at 2:44 PM, Mahesh Dananjaya < >>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>> It is very interesting that i have found that somethings that >>>>>>>>>>>> can make use of our work. In the cep 4.0 documentation there is a >>>>>>>>>>>> Custom >>>>>>>>>>>> Stream Processor Extention program [1]. There is a example of >>>>>>>>>>>> LinearRegressionStreamProcessor [1] and also i saw >>>>>>>>>>>> private int batchSize = 1000000000; i am going through this >>>>>>>>>>>> one. >>>>>>>>>>>> Please check whether we can use. WIll there be any >>>>>>>>>>>> compatibility or support issue? >>>>>>>>>>>> regards, >>>>>>>>>>>> Mahesh. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension >>>>>>>>>>>> >>>>>>>>>>>> On Sat, May 21, 2016 at 11:52 AM, Mahesh Dananjaya < >>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi maheshakya, >>>>>>>>>>>>> anyway how can test any siddhi extention after write it >>>>>>>>>>>>> without integrating it to cep.can you please explain me the >>>>>>>>>>>>> procedure. i am >>>>>>>>>>>>> referring to [1] [2] [3] [4]. thank you. >>>>>>>>>>>>> BR, >>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>>>>>> https://docs.wso2.com/display/CEP310/Writing+Extensions+to+Siddhi >>>>>>>>>>>>> [2] >>>>>>>>>>>>> https://docs.wso2.com/display/CEP310/Writing+a+Custom+Function >>>>>>>>>>>>> [3] >>>>>>>>>>>>> https://docs.wso2.com/display/CEP310/Writing+a+Custom+Window >>>>>>>>>>>>> [4] >>>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+Extensions+to+Siddhi >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, May 19, 2016 at 12:08 PM, Mahesh Dananjaya < >>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>> thank you for the feedback. I have add data-sets into repo. >>>>>>>>>>>>>> data-sets/lr. I am all right with next week.Now i am writing >>>>>>>>>>>>>> some examples >>>>>>>>>>>>>> to collect samples and build mini batches and run the algorithms >>>>>>>>>>>>>> on those >>>>>>>>>>>>>> mini-batches. thank you. will add those into repo soon.I am >>>>>>>>>>>>>> still working >>>>>>>>>>>>>> on that siddhi extention.i will let you know the progress. >>>>>>>>>>>>>> BR, >>>>>>>>>>>>>> mahesh. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, May 19, 2016 at 11:10 AM, Maheshakya Wijewardena < >>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I've look into your code sample of streaming linear >>>>>>>>>>>>>>> regression. Looks good to me, apart from few issues in coding >>>>>>>>>>>>>>> practices >>>>>>>>>>>>>>> which we can improve when you're doing the implementations in >>>>>>>>>>>>>>> carbon-ml and >>>>>>>>>>>>>>> during the code reviews. You are using a set of files as >>>>>>>>>>>>>>> mini-batches of >>>>>>>>>>>>>>> data, right? Can you also send us the datasets you've been >>>>>>>>>>>>>>> using. I'd like >>>>>>>>>>>>>>> to run this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> does that cep problem is now all right that we were trying >>>>>>>>>>>>>>>> to fix. I am still using those pre-build versions. If so i can >>>>>>>>>>>>>>>> merge with >>>>>>>>>>>>>>>> the latest one. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'll check this and let you know. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can we arrange a meeting (preferably in WSO2 offices) in >>>>>>>>>>>>>>> next week with ML team members as well. Coding period begins on >>>>>>>>>>>>>>> next >>>>>>>>>>>>>>> Monday, so it's better to get overall feedback from others and >>>>>>>>>>>>>>> discuss more >>>>>>>>>>>>>>> about the project. Let me know convenient time slots for you. >>>>>>>>>>>>>>> I'll arrange >>>>>>>>>>>>>>> a meeting with ML team. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, May 18, 2016 at 9:53 AM, Mahesh Dananjaya < >>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>> Ok. I will check it.you have sent me those relevant >>>>>>>>>>>>>>>> references and i am working on that thing.thank you. does that >>>>>>>>>>>>>>>> cep problem >>>>>>>>>>>>>>>> is now all right that we were trying to fix. I am still using >>>>>>>>>>>>>>>> those >>>>>>>>>>>>>>>> pre-build versions. If so i can merge with the latest >>>>>>>>>>>>>>>> one.thanks. >>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, May 18, 2016 at 9:44 AM, Maheshakya Wijewardena < >>>>>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You don't actually have to implement anything in spark >>>>>>>>>>>>>>>>> streaming. Try to understand how streaming data is handled in >>>>>>>>>>>>>>>>> and the >>>>>>>>>>>>>>>>> specifics of the underlying algorithms in streaming. >>>>>>>>>>>>>>>>> What we want to do is having the similar algorithms that >>>>>>>>>>>>>>>>> support CEP event streams with siddhi. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, May 18, 2016 at 9:38 AM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>> Did you check the repo. I will add recent works today.And >>>>>>>>>>>>>>>>>> also i was going through the Java docs related to spark >>>>>>>>>>>>>>>>>> streaming work. It >>>>>>>>>>>>>>>>>> is with that scala API. thank you. >>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, May 17, 2016 at 10:11 AM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>> I have gone through the Java Docs and run some of the >>>>>>>>>>>>>>>>>>> Spark examples on spark shell which are paramount improtant >>>>>>>>>>>>>>>>>>> for our work. >>>>>>>>>>>>>>>>>>> Then i have been writing my codes to check the Linear >>>>>>>>>>>>>>>>>>> regression, K means >>>>>>>>>>>>>>>>>>> for streaming. please check my git repo [1]. I think now i >>>>>>>>>>>>>>>>>>> have to ask on >>>>>>>>>>>>>>>>>>> dev regarding the capturing event streams for our work. I >>>>>>>>>>>>>>>>>>> will update the >>>>>>>>>>>>>>>>>>> recent things on git. check the park-example directory for >>>>>>>>>>>>>>>>>>> java. examples >>>>>>>>>>>>>>>>>>> run on git shell is not included there. In my case i think >>>>>>>>>>>>>>>>>>> i have to build >>>>>>>>>>>>>>>>>>> mini batches from data streams that comes as individual >>>>>>>>>>>>>>>>>>> samples. Now i am >>>>>>>>>>>>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>>>>>>>>>>>> streams.thank you. >>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, May 17, 2016 at 10:10 AM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>> I have gone through the Java Docs and run some of the >>>>>>>>>>>>>>>>>>>> Spark examples on spark shell which are paramount >>>>>>>>>>>>>>>>>>>> improtant for our work. >>>>>>>>>>>>>>>>>>>> Then i have been writing my codes to check the Linear >>>>>>>>>>>>>>>>>>>> regression, K means >>>>>>>>>>>>>>>>>>>> for streaming. please check my git repo [1]. I think now i >>>>>>>>>>>>>>>>>>>> have to ask on >>>>>>>>>>>>>>>>>>>> dev regarding the capturing event streams for our work. I >>>>>>>>>>>>>>>>>>>> will update the >>>>>>>>>>>>>>>>>>>> recent things on git. check the park-example directory for >>>>>>>>>>>>>>>>>>>> java. examples >>>>>>>>>>>>>>>>>>>> run on git shell is not included there. In my case i think >>>>>>>>>>>>>>>>>>>> i have to build >>>>>>>>>>>>>>>>>>>> mini batches from data streams that comes as individual >>>>>>>>>>>>>>>>>>>> samples. Now i am >>>>>>>>>>>>>>>>>>>> working on some coding to collect mini batches from data >>>>>>>>>>>>>>>>>>>> streams.thank you. >>>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Mon, May 16, 2016 at 1:19 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>> thank you. i will update the repo today.thank you.i >>>>>>>>>>>>>>>>>>>>> changed the carbon ml siddhi extention and see how the >>>>>>>>>>>>>>>>>>>>> changes are >>>>>>>>>>>>>>>>>>>>> effecting. i will update the progress as soon as >>>>>>>>>>>>>>>>>>>>> possible.thank you. i had >>>>>>>>>>>>>>>>>>>>> some problem in spark mllib dependency. i was fixing that. >>>>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>> p.s: do i need to maintain a blog? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Mon, May 16, 2016 at 10:02 AM, Maheshakya >>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Sorry for replying late. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thank you for the update. I believe you have done >>>>>>>>>>>>>>>>>>>>>> some implementations with with Spark MLLIb algorithms in >>>>>>>>>>>>>>>>>>>>>> streaming fashion >>>>>>>>>>>>>>>>>>>>>> as we have discussed. If so, can you please share your >>>>>>>>>>>>>>>>>>>>>> code in a Github >>>>>>>>>>>>>>>>>>>>>> repo. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Now i want to implements some machine learning >>>>>>>>>>>>>>>>>>>>>>> algorithms with importing mllib and want to run within >>>>>>>>>>>>>>>>>>>>>>> your code base >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> For the moment you can try out editing the same class >>>>>>>>>>>>>>>>>>>>>> PredictStreamProcessor in the siddhi extension in >>>>>>>>>>>>>>>>>>>>>> carbon-ml. Later we will >>>>>>>>>>>>>>>>>>>>>> add this separately. You should be able to add >>>>>>>>>>>>>>>>>>>>>> org.apache.spark.mllib. >>>>>>>>>>>>>>>>>>>>>> classes to there. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> And i want to see how event streams are coming from >>>>>>>>>>>>>>>>>>>>>>> cep. As i think it is not in a RDD format since it is >>>>>>>>>>>>>>>>>>>>>>> arriving as the >>>>>>>>>>>>>>>>>>>>>>> individual samples. I will send a email to dev asking >>>>>>>>>>>>>>>>>>>>>>> about how to get the >>>>>>>>>>>>>>>>>>>>>>> streams. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Please pay attention to length[1] and lengthbatch[1] >>>>>>>>>>>>>>>>>>>>>> inbuilt windows in siddhi. What you need to write are >>>>>>>>>>>>>>>>>>>>>> functions similar to >>>>>>>>>>>>>>>>>>>>>> a custom aggregate function[2]. >>>>>>>>>>>>>>>>>>>>>> When you send the email to dev list, explain your >>>>>>>>>>>>>>>>>>>>>> requirement. You need to get a set of event with from a >>>>>>>>>>>>>>>>>>>>>> stream with a >>>>>>>>>>>>>>>>>>>>>> specified window size (number of events). Then build a >>>>>>>>>>>>>>>>>>>>>> model within that >>>>>>>>>>>>>>>>>>>>>> function. You also need to retain the data (learned >>>>>>>>>>>>>>>>>>>>>> weights, cluster >>>>>>>>>>>>>>>>>>>>>> centers, etc.) from the previous window to use in the >>>>>>>>>>>>>>>>>>>>>> current window. Ask >>>>>>>>>>>>>>>>>>>>>> what can be the most suitable option for this among the >>>>>>>>>>>>>>>>>>>>>> set of siddhi >>>>>>>>>>>>>>>>>>>>>> extensions given. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Inbuilt+Windows#InbuiltWindows-lengthlength >>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/CEP400/Writing+a+Custom+Aggregate+Function >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Wed, May 11, 2016 at 1:43 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> ---------- Forwarded message ---------- >>>>>>>>>>>>>>>>>>>>>>> From: Mahesh Dananjaya <dananjayamah...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>> Date: Wed, May 11, 2016 at 1:43 PM >>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [Dev] GSOC2016: [ML][CEP] Predictive >>>>>>>>>>>>>>>>>>>>>>> analytic with online data for WSO2 Machine Learner >>>>>>>>>>>>>>>>>>>>>>> To: Maheshakya Wijewardena <mahesha...@wso2.com> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>>>> sorry for not updating. I did what you wanted me to >>>>>>>>>>>>>>>>>>>>>>> do. I checked the code base and train functions. I went >>>>>>>>>>>>>>>>>>>>>>> through those java >>>>>>>>>>>>>>>>>>>>>>> docs. I went through the carbon-ml current >>>>>>>>>>>>>>>>>>>>>>> implementation of LG and K-Mean. >>>>>>>>>>>>>>>>>>>>>>> And i had Apache Spark and i tried with several >>>>>>>>>>>>>>>>>>>>>>> examples. Now i want to >>>>>>>>>>>>>>>>>>>>>>> implements some machine learning algorithms with >>>>>>>>>>>>>>>>>>>>>>> importing mllib and want >>>>>>>>>>>>>>>>>>>>>>> to run within your code base. Can you help me with that. >>>>>>>>>>>>>>>>>>>>>>> And i want to see how event streams are coming from >>>>>>>>>>>>>>>>>>>>>>> cep. As i think it is not in a RDD format since it is >>>>>>>>>>>>>>>>>>>>>>> arriving as the >>>>>>>>>>>>>>>>>>>>>>> individual samples. I will send a email to dev asking >>>>>>>>>>>>>>>>>>>>>>> about how to get the >>>>>>>>>>>>>>>>>>>>>>> streams. I debugged many of those functions in the code >>>>>>>>>>>>>>>>>>>>>>> base. So need >>>>>>>>>>>>>>>>>>>>>>> further instructions to proceed.thank you. >>>>>>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Wed, May 11, 2016 at 10:32 AM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Any update on your progress? >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 4, 2016 at 8:35 PM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> is that "Put break points in train methods in >>>>>>>>>>>>>>>>>>>>>>>>>> Linear Regression class" means the spark/algorithms/ >>>>>>>>>>>>>>>>>>>>>>>>>> LinearRegrassion.java >>>>>>>>>>>>>>>>>>>>>>>>>> class in the org.wso2.carbon.ml.core? is that the >>>>>>>>>>>>>>>>>>>>>>>>>> correct file? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Yes, this is the correct place. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> You can refer to spark programming guide[1][2] as >>>>>>>>>>>>>>>>>>>>>>>>> well as our ML code base when you try those >>>>>>>>>>>>>>>>>>>>>>>>> algorithms out. Please try to >>>>>>>>>>>>>>>>>>>>>>>>> do rough implementations of the streaming versions of >>>>>>>>>>>>>>>>>>>>>>>>> linear regression, >>>>>>>>>>>>>>>>>>>>>>>>> logistic regression and k-means clustering as we have >>>>>>>>>>>>>>>>>>>>>>>>> discussed in the >>>>>>>>>>>>>>>>>>>>>>>>> proposal in plain Java. It's better if you can create >>>>>>>>>>>>>>>>>>>>>>>>> a git repo and share >>>>>>>>>>>>>>>>>>>>>>>>> your code once you have made some progress. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Were you able debug and understand the flow of the >>>>>>>>>>>>>>>>>>>>>>>>> ML siddhi extension? I hope you haven't encountered >>>>>>>>>>>>>>>>>>>>>>>>> more errors after >>>>>>>>>>>>>>>>>>>>>>>>> switching the released version of CEP. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Is this Friday okay for you? Afternoon at 2:00 pm? >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>> http://spark.apache.org/docs/latest/programming-guide.html >>>>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>>>> http://spark.apache.org/docs/latest/mllib-guide.html >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>>>>>>> I have been looking into some algorithms related >>>>>>>>>>>>>>>>>>>>>>>>>> to stochastic gradient descent based >>>>>>>>>>>>>>>>>>>>>>>>>> algorithms.anything i should focus >>>>>>>>>>>>>>>>>>>>>>>>>> please let me know.Ans also i will be available for >>>>>>>>>>>>>>>>>>>>>>>>>> calling this week and >>>>>>>>>>>>>>>>>>>>>>>>>> next week.thank you. >>>>>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 5:05 PM, Mahesh Dananjaya >>>>>>>>>>>>>>>>>>>>>>>>>> <dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>>>>>>>> thank you.that's good. i have been trying to fix >>>>>>>>>>>>>>>>>>>>>>>>>>> that for couple of days. please inform me when it >>>>>>>>>>>>>>>>>>>>>>>>>>> will be fixed.now i have >>>>>>>>>>>>>>>>>>>>>>>>>>> been testing the ML algorithms and trying to >>>>>>>>>>>>>>>>>>>>>>>>>>> identify the flow and the >>>>>>>>>>>>>>>>>>>>>>>>>>> hierarchy. is that "Put break points in train >>>>>>>>>>>>>>>>>>>>>>>>>>> methods in Linear Regression >>>>>>>>>>>>>>>>>>>>>>>>>>> class" means the spark/algorithms/ >>>>>>>>>>>>>>>>>>>>>>>>>>> LinearRegrassion.java class in the >>>>>>>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core? is that the correct file? >>>>>>>>>>>>>>>>>>>>>>>>>>> And also i am planning to write some programs to >>>>>>>>>>>>>>>>>>>>>>>>>>> use apache spark mllib algorithms. and i refer to >>>>>>>>>>>>>>>>>>>>>>>>>>> [1] and some wso2 >>>>>>>>>>>>>>>>>>>>>>>>>>> documentations to get some idea about ML >>>>>>>>>>>>>>>>>>>>>>>>>>> structure.thank you. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> [1]nirmalfdo.blogspot.com >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 4:36 PM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I have checked. It seems the issue you have >>>>>>>>>>>>>>>>>>>>>>>>>>>> encountered is cause only in the current >>>>>>>>>>>>>>>>>>>>>>>>>>>> development branch of the >>>>>>>>>>>>>>>>>>>>>>>>>>>> product-cep. It doesn't identify the ML siddhi >>>>>>>>>>>>>>>>>>>>>>>>>>>> extension as an extension. >>>>>>>>>>>>>>>>>>>>>>>>>>>> ML siddhi extension works fine in the latest >>>>>>>>>>>>>>>>>>>>>>>>>>>> release of CEP (4.1.0) [1]. >>>>>>>>>>>>>>>>>>>>>>>>>>>> Until we figure out the reason and come up with >>>>>>>>>>>>>>>>>>>>>>>>>>>> a solution, can you use the latest CEP release for >>>>>>>>>>>>>>>>>>>>>>>>>>>> your work. It's fine to >>>>>>>>>>>>>>>>>>>>>>>>>>>> use that since you haven't started actual >>>>>>>>>>>>>>>>>>>>>>>>>>>> development yet. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>> http://wso2.com/products/complex-event-processor/ >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 3:19 PM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is is vital to use those local repo in my >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upcoming implementation? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yes. The remote p2-repo contains the p2-repos >>>>>>>>>>>>>>>>>>>>>>>>>>>>> of released versions. What you have to develop on >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is the current master of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the carbon-ml and product-ml. You can try out >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the modification I have >>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggested. In the meantime, I'll verify whether >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the current repos are >>>>>>>>>>>>>>>>>>>>>>>>>>>>> working as expected. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's great that you have started looking at >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the implementation of linear regression as well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Put break points in train >>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods in LinearRegression class. This is being >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used when you run linear >>>>>>>>>>>>>>>>>>>>>>>>>>>>> regression from UI. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I can see some comments left behind for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> streaming algo as well.thank you >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> You may be referring to the linear regression >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with SGD model. Here, there's no retraining with >>>>>>>>>>>>>>>>>>>>>>>>>>>>> streaming data involved. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The SGD with minibatches is used to train the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> model with the data set only >>>>>>>>>>>>>>>>>>>>>>>>>>>>> once. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> What you have to do is create a similar >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mechanism to involve streaming data and retrain >>>>>>>>>>>>>>>>>>>>>>>>>>>>> models. We will get to that >>>>>>>>>>>>>>>>>>>>>>>>>>>>> part once you get comfortable with siddhi >>>>>>>>>>>>>>>>>>>>>>>>>>>>> extensions. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> BTW, is it possible for you to join a call on >>>>>>>>>>>>>>>>>>>>>>>>>>>>> this Friday or in the next week. We'll try to >>>>>>>>>>>>>>>>>>>>>>>>>>>>> resolve your current issues >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and discuss further on project. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 1:03 PM, Mahesh >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dananjaya <dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi maheshakya, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is it ok to go with p2 repo at >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://product-dist.wso2.com/p2/carbon/releases/wilkes/features/ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather than the P2-repo at >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> product-ml/modules/p2-profile/target/p2-repo in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> local repo.What is the impact?.Is is vital to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> use those local repo in my >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upcoming implementation?so i was trying to give >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> remote p2 repo to cep built >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by source and trying to debug the cep ml >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extension and got the same error >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as yesterday. But pre-built product is working >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine. Therefore now i am >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trying as you described in the last email. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And also i am trying to debug the carbon-ml >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.core by putting some break >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> point in the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> spark/algorithms/Linear Regression. I am trying >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to trigger it with >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> product-ml project with data set. Does that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Linear Regression also in UI is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consuming that spark algorithms or is it in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> another place? I can see some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments left behind for streaming algo as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well.thank you. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 3, 2016 at 9:35 AM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The earlier error you have mentioned may >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> occur due to the incompatible Siddhi versions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in ML p2-repo and CEP, when >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you add the p2-repo from prodcut-ml you built. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Current siddhi version in product-cep is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3.0.6-SNAPSHOT[1], but in ML it's 3.0.2. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try changing the siddhi.version in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> carbon-ml/pom.xml to 3.0.6-SNAPSHOT, build >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> carbon-ml, then build product-ml >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> again. After this, add p2-repo as a local >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> repository again to fresh CEP >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pack and try it out. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, May 2, 2016 at 7:02 PM, Mahesh >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dananjaya <dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now i remotely debug the CEP extension for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ML Prediction. What i did was, i have all the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pre-build version of CEP and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ML. Therefore i used that built CEP and did >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same thing that i was doing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with the source code. I think the only change >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i did was install those >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> packages from remote p2 repo. This was work >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine and i debugged the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> carbon-ml org.wso2.carbon.ml.siddhi.extension >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as described in the [1]. So >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now i have to try same thing with the build by >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source content.thank you. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> BR, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mahesh >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 5:49 PM, Maheshakya >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wijewardena <mahesha...@wso2.com> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Congratulations and welcome to GSoC 2016. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> You did a great job in preparing the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal. Now it's time to dig deep >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and get started with the project. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> First of all you need to familiarize with >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the code base. We have agreed to implement >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this with CEP event streams. We >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already have a CEP extension for predictions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1][2]. Go through this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation and familiarize your self with >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that. You need to understand >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Even streams are consumed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. predictions are made from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> individual event >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Results are sent back >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Get WSO2 ML and CEP sources (You may use >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest released version of CEP) and build the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> products. Get both >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> carbon-ml[3] and product-ml[4] masters and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> create new branches for your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> work from masters. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After you build the products, you may need >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to do remote debugging[5] to understand the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow. So please follow an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example of real time prediction with ML with >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> debugging and get some idea. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The component you need to debug is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> org.wso2.carbon.ml.siddhi.extension. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Next tasks would be implementing online >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> learning algorithms in plain java with spark >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ml lib and integrating those >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to ML. We also need to come up with a proper >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and detailed architecture to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> employ those algorithms in ML. Getting >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar with the aforementioned >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sections would give you some insight on how >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this should be implemented. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So please try to get a quick grasp then >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can start the implementation. Let us know >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if you have any questions or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you get stuck somewhere. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, please always add WSO2 developer's >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list as well when you communicate with us >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding the project so that you >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can get opinions and feedback from others as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/WSO2+CEP+Extension+for+ML+Predictions#WSO2CEPExtensionforMLPredictions-Siddhisyntaxfortheextension >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/wso2/carbon-ml/tree/master/components/extensions/org.wso2.carbon.ml.siddhi.extension >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [3] https://github.com/wso2/carbon-ml >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [4] https://github.com/wso2/product-ml >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [5] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://dzone.com/articles/how-debug-wso2-carbon-kernel >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 25, 2016 at 3:33 PM, Mahesh >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Dananjaya <dananjayamah...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thank you for accepting my GSOC 2016 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and i am looking forward for the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> further instruction and project >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> continuation. thank you very much. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>> mahesha...@wso2.com >>>>>>>>>> +94711228855 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>> mahesha...@wso2.com >>>>>>>>> +94711228855 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pruthuvi Maheshakya Wijewardena >>>>> mahesha...@wso2.com >>>>> +94711228855 >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Senior Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
