Okay got it! :) @supun : If you don't mind can you share the architecture diagram. I'm bit interested in how it's done. how can we introduce new algorithms to the component?
Thanks, Rajith On Thu, Jul 24, 2014 at 10:33 AM, Supun Sethunga <[email protected]> wrote: > Hi Rajith, > > The service is a stand-alone carbon component, and not a UDAF. Hive will > only be used to retrieve and store the data which will eventually used by > the service. > > Regards, > Supun > > > On Thu, Jul 24, 2014 at 9:28 AM, Rajith Siriwardena <[email protected]> > wrote: > >> Hi Supun, >> >> Are you using UDAFs for the purpose? Can explain a bit how it's done? >> >> >> On Wed, Jul 23, 2014 at 5:30 PM, Supun Sethunga <[email protected]> wrote: >> >>> Hi, >>> >>> The back-end service for the $subject has been implemented. For the time >>> being, it supports three algorithms (Naive Bayes, Logistic Regression and >>> Multilayer Perceptrons). The service accepts a JSON object having the input >>> data file path, output path where the created the model should be saved, >>> the algorithm to be used for modeling (one of the above three algos), and >>> the hyper-parameters and their values. >>> >>> Next goal would be to develop a carbon UI component, which allows a >>> user to call the above service, in a form of a wizard. The wizard would >>> provide a step-by-step guide to the user to create a ML model for any set >>> of data. >>> >>> Appreciate any suggestions. >>> >>> Regards, >>> Supun >>> >>> >>> On Fri, Jul 4, 2014 at 7:39 AM, Srinath Perera <[email protected]> wrote: >>> >>>> Anjana and myself discussed. >>>> >>>> UDFs does not work. UDFs take an event as the input while mahout takes >>>> a complete dataset. >>>> >>>> May be we can use UDAF (A = Aggregate), but has to explore more. >>>> >>>> >>>> On Tue, Jul 1, 2014 at 10:03 AM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> I see, sure, I was thinking of doing all the operations, including the >>>>> training operations using an UDF. Will come and meet you. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> >>>>> On Tue, Jul 1, 2014 at 9:56 AM, Srinath Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> No, we need to get the data, preprocess them using hive, and send all >>>>>> the data (not 1-2 values, rather say 10 millions values) to training >>>>>> phase. >>>>>> Lets chat f2f a bit. >>>>>> >>>>>> --Srinath >>>>>> >>>>>> >>>>>> On Tue, Jul 1, 2014 at 6:24 AM, Anjana Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I was simply thinking, the UDF could directly mapped to some basic >>>>>>> Mahout operation it implements, and the input/output should be given as >>>>>>> parameters to the UDF, so probably, we can publish some input data >>>>>>> beforehand to Cassandra etc.. and give the location of that data to the >>>>>>> UDF, and the UDF will, as it is called, create the map/reduce jobs and >>>>>>> execute. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> >>>>>>> On Tue, Jul 1, 2014 at 9:18 AM, Srinath Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> +1 we wanted to explore that more. >>>>>>>> >>>>>>>> However, It is not a simple UDF as this is a stateful op where we >>>>>>>> feed lot of data and start a separate map reduce process. Anjana, do >>>>>>>> you >>>>>>>> have any thoughts on how it can be done? >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 1, 2014 at 5:37 AM, Anjana Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm just wondering if we have any way to integrate this to Hive >>>>>>>>> itself (UDF?), to get results of an ML algorithm run, to a result >>>>>>>>> there. A >>>>>>>>> similar scenario is possible in Shark/MLlib integration. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jun 30, 2014 at 12:28 PM, Supun Sethunga <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Im working on the $subject, and the objective is to apply Machine >>>>>>>>>> Learning algorithms on the data stored by WSO2 BAM. Apache Mahout >>>>>>>>>> will be >>>>>>>>>> used as the ML tool, for this purpose. >>>>>>>>>> >>>>>>>>>> As per the discussion I had With Srinath, the procedure for >>>>>>>>>> $subject would be: >>>>>>>>>> >>>>>>>>>> - Test a Machine Learning algorithm using Mahout libraries >>>>>>>>>> within Java. >>>>>>>>>> - Implement a RESTful service which provides the above >>>>>>>>>> functionality. >>>>>>>>>> - Since Mahout also uses Hadoop, the above service can send >>>>>>>>>> Map Reduce Jobs to the Hadoop built inside the BAM. >>>>>>>>>> - Deploy the service as a Carbon Component on WSO2 BAM. >>>>>>>>>> >>>>>>>>>> The first step is completed for now. >>>>>>>>>> Any feedback is highly appreciated. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Supun >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Supun Sethunga* >>>>>>>>>> Software Engineer >>>>>>>>>> WSO2, Inc. >>>>>>>>>> lean | enterprise | middleware >>>>>>>>>> Mobile : +94 716546324 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Senior Technical Lead >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Director, Research, WSO2 Inc. >>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>> Member, Apache Software Foundation >>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ============================ >>>>>> Director, Research, WSO2 Inc. >>>>>> Visiting Faculty, University of Moratuwa >>>>>> Member, Apache Software Foundation >>>>>> Research Scientist, Lanka Software Foundation >>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>> Site: http://people.apache.org/~hemapani/ >>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>> Phone: 0772360902 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Senior Technical Lead >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> ============================ >>>> Director, Research, WSO2 Inc. >>>> Visiting Faculty, University of Moratuwa >>>> Member, Apache Software Foundation >>>> Research Scientist, Lanka Software Foundation >>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> Phone: 0772360902 >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Rajith Siriwardana* >> Software Engineer >> WSO2 Inc. ; http://wso2.com >> *lean. enterprise. middleware* >> >> ------------------------------------------------------ >> *http://people.apache.org/~siriwardana >> <http://people.apache.org/~siriwardana>* >> > > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Rajith Siriwardana* Software Engineer WSO2 Inc. ; http://wso2.com *lean. enterprise. middleware* ------------------------------------------------------ *http://people.apache.org/~siriwardana <http://people.apache.org/~siriwardana>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
