Can we chat f2f sometime, would like to understand the scenarios?
On Fri, Jul 25, 2014 at 9:30 AM, Rajith Siriwardena <[email protected]> wrote: > Thanks for the clarification. I was thinking of adopting this to improve > WSO2 Cloud Monitor by introducing ML to the system. Currently we are using > UDFs and UDAFs for the purpose. :) > > Thanks, > Rajith > > > On Fri, Jul 25, 2014 at 9:16 AM, Srinath Perera <[email protected]> wrote: > >> Currently, you have to edit code the add new algorithms. We will make it >> configurable eventually, but at this stage we do not need a framework IMHO. >> >> --Srinath >> >> >> On Thu, Jul 24, 2014 at 7:46 PM, Rajith Siriwardena <[email protected]> >> wrote: >> >>> >>> Okay got it! :) >>> >>> @supun : If you don't mind can you share the architecture diagram. I'm >>> bit interested in how it's done. how can we introduce new algorithms to the >>> component? >>> >>> Thanks, >>> Rajith >>> >>> >>> On Thu, Jul 24, 2014 at 10:33 AM, Supun Sethunga <[email protected]> >>> wrote: >>> >>>> Hi Rajith, >>>> >>>> The service is a stand-alone carbon component, and not a UDAF. Hive >>>> will only be used to retrieve and store the data which will eventually used >>>> by the service. >>>> >>>> Regards, >>>> Supun >>>> >>>> >>>> On Thu, Jul 24, 2014 at 9:28 AM, Rajith Siriwardena <[email protected]> >>>> wrote: >>>> >>>>> Hi Supun, >>>>> >>>>> Are you using UDAFs for the purpose? Can explain a bit how it's done? >>>>> >>>>> >>>>> On Wed, Jul 23, 2014 at 5:30 PM, Supun Sethunga <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> The back-end service for the $subject has been implemented. For the >>>>>> time being, it supports three algorithms (Naive Bayes, Logistic >>>>>> Regression >>>>>> and Multilayer Perceptrons). The service accepts a JSON object having the >>>>>> input data file path, output path where the created the model should be >>>>>> saved, the algorithm to be used for modeling (one of the above three >>>>>> algos), and the hyper-parameters and their values. >>>>>> >>>>>> Next goal would be to develop a carbon UI component, which allows a >>>>>> user to call the above service, in a form of a wizard. The wizard would >>>>>> provide a step-by-step guide to the user to create a ML model for any set >>>>>> of data. >>>>>> >>>>>> Appreciate any suggestions. >>>>>> >>>>>> Regards, >>>>>> Supun >>>>>> >>>>>> >>>>>> On Fri, Jul 4, 2014 at 7:39 AM, Srinath Perera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Anjana and myself discussed. >>>>>>> >>>>>>> UDFs does not work. UDFs take an event as the input while mahout >>>>>>> takes a complete dataset. >>>>>>> >>>>>>> May be we can use UDAF (A = Aggregate), but has to explore more. >>>>>>> >>>>>>> >>>>>>> On Tue, Jul 1, 2014 at 10:03 AM, Anjana Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I see, sure, I was thinking of doing all the operations, including >>>>>>>> the training operations using an UDF. Will come and meet you. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 1, 2014 at 9:56 AM, Srinath Perera <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> No, we need to get the data, preprocess them using hive, and send >>>>>>>>> all the data (not 1-2 values, rather say 10 millions values) to >>>>>>>>> training >>>>>>>>> phase. Lets chat f2f a bit. >>>>>>>>> >>>>>>>>> --Srinath >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jul 1, 2014 at 6:24 AM, Anjana Fernando <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I was simply thinking, the UDF could directly mapped to some >>>>>>>>>> basic Mahout operation it implements, and the input/output should be >>>>>>>>>> given >>>>>>>>>> as parameters to the UDF, so probably, we can publish some input data >>>>>>>>>> beforehand to Cassandra etc.. and give the location of that data to >>>>>>>>>> the >>>>>>>>>> UDF, and the UDF will, as it is called, create the map/reduce jobs >>>>>>>>>> and >>>>>>>>>> execute. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Anjana. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 1, 2014 at 9:18 AM, Srinath Perera <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> +1 we wanted to explore that more. >>>>>>>>>>> >>>>>>>>>>> However, It is not a simple UDF as this is a stateful op where >>>>>>>>>>> we feed lot of data and start a separate map reduce process. >>>>>>>>>>> Anjana, do you >>>>>>>>>>> have any thoughts on how it can be done? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 1, 2014 at 5:37 AM, Anjana Fernando <[email protected] >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I'm just wondering if we have any way to integrate this to Hive >>>>>>>>>>>> itself (UDF?), to get results of an ML algorithm run, to a result >>>>>>>>>>>> there. A >>>>>>>>>>>> similar scenario is possible in Shark/MLlib integration. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Anjana. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jun 30, 2014 at 12:28 PM, Supun Sethunga < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Im working on the $subject, and the objective is to apply >>>>>>>>>>>>> Machine Learning algorithms on the data stored by WSO2 BAM. >>>>>>>>>>>>> Apache Mahout >>>>>>>>>>>>> will be used as the ML tool, for this purpose. >>>>>>>>>>>>> >>>>>>>>>>>>> As per the discussion I had With Srinath, the procedure for >>>>>>>>>>>>> $subject would be: >>>>>>>>>>>>> >>>>>>>>>>>>> - Test a Machine Learning algorithm using Mahout libraries >>>>>>>>>>>>> within Java. >>>>>>>>>>>>> - Implement a RESTful service which provides the above >>>>>>>>>>>>> functionality. >>>>>>>>>>>>> - Since Mahout also uses Hadoop, the above service can >>>>>>>>>>>>> send Map Reduce Jobs to the Hadoop built inside the BAM. >>>>>>>>>>>>> - Deploy the service as a Carbon Component on WSO2 BAM. >>>>>>>>>>>>> >>>>>>>>>>>>> The first step is completed for now. >>>>>>>>>>>>> Any feedback is highly appreciated. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Supun >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Supun Sethunga* >>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>>> lean | enterprise | middleware >>>>>>>>>>>>> Mobile : +94 716546324 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>>> Senior Technical Lead >>>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> ============================ >>>>>>>>>>> Director, Research, WSO2 Inc. >>>>>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>>>>> Member, Apache Software Foundation >>>>>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Anjana Fernando* >>>>>>>>>> Senior Technical Lead >>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>> lean . enterprise . middleware >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> ============================ >>>>>>>>> Director, Research, WSO2 Inc. >>>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>>> Member, Apache Software Foundation >>>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>> Phone: 0772360902 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Senior Technical Lead >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ============================ >>>>>>> Director, Research, WSO2 Inc. >>>>>>> Visiting Faculty, University of Moratuwa >>>>>>> Member, Apache Software Foundation >>>>>>> Research Scientist, Lanka Software Foundation >>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>> Phone: 0772360902 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Supun Sethunga* >>>>>> Software Engineer >>>>>> WSO2, Inc. >>>>>> lean | enterprise | middleware >>>>>> Mobile : +94 716546324 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Rajith Siriwardana* >>>>> Software Engineer >>>>> WSO2 Inc. ; http://wso2.com >>>>> *lean. enterprise. middleware* >>>>> >>>>> ------------------------------------------------------ >>>>> *http://people.apache.org/~siriwardana >>>>> <http://people.apache.org/~siriwardana>* >>>>> >>>> >>>> >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >>> >>> -- >>> *Rajith Siriwardana* >>> Software Engineer >>> WSO2 Inc. ; http://wso2.com >>> *lean. enterprise. middleware* >>> >>> ------------------------------------------------------ >>> *http://people.apache.org/~siriwardana >>> <http://people.apache.org/~siriwardana>* >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> ============================ >> Director, Research, WSO2 Inc. >> Visiting Faculty, University of Moratuwa >> Member, Apache Software Foundation >> Research Scientist, Lanka Software Foundation >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Rajith Siriwardana* > Software Engineer > WSO2 Inc. ; http://wso2.com > *lean. enterprise. middleware* > > ------------------------------------------------------ > *http://people.apache.org/~siriwardana > <http://people.apache.org/~siriwardana>* > -- ============================ Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
