Above service does not integrate ML with Hive Language. UDAF does not works as well. Linkedin guys have done it as a HIVE store, but it is a hack IMO.
On Thu, Jul 24, 2014 at 9:28 AM, Rajith Siriwardena <[email protected]> wrote: > Hi Supun, > > Are you using UDAFs for the purpose? Can explain a bit how it's done? > > > On Wed, Jul 23, 2014 at 5:30 PM, Supun Sethunga <[email protected]> wrote: > >> Hi, >> >> The back-end service for the $subject has been implemented. For the time >> being, it supports three algorithms (Naive Bayes, Logistic Regression and >> Multilayer Perceptrons). The service accepts a JSON object having the input >> data file path, output path where the created the model should be saved, >> the algorithm to be used for modeling (one of the above three algos), and >> the hyper-parameters and their values. >> >> Next goal would be to develop a carbon UI component, which allows a user >> to call the above service, in a form of a wizard. The wizard would provide >> a step-by-step guide to the user to create a ML model for any set of >> data. >> >> Appreciate any suggestions. >> >> Regards, >> Supun >> >> >> On Fri, Jul 4, 2014 at 7:39 AM, Srinath Perera <[email protected]> wrote: >> >>> Anjana and myself discussed. >>> >>> UDFs does not work. UDFs take an event as the input while mahout takes a >>> complete dataset. >>> >>> May be we can use UDAF (A = Aggregate), but has to explore more. >>> >>> >>> On Tue, Jul 1, 2014 at 10:03 AM, Anjana Fernando <[email protected]> >>> wrote: >>> >>>> I see, sure, I was thinking of doing all the operations, including the >>>> training operations using an UDF. Will come and meet you. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> >>>> On Tue, Jul 1, 2014 at 9:56 AM, Srinath Perera <[email protected]> >>>> wrote: >>>> >>>>> No, we need to get the data, preprocess them using hive, and send all >>>>> the data (not 1-2 values, rather say 10 millions values) to training >>>>> phase. >>>>> Lets chat f2f a bit. >>>>> >>>>> --Srinath >>>>> >>>>> >>>>> On Tue, Jul 1, 2014 at 6:24 AM, Anjana Fernando <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I was simply thinking, the UDF could directly mapped to some basic >>>>>> Mahout operation it implements, and the input/output should be given as >>>>>> parameters to the UDF, so probably, we can publish some input data >>>>>> beforehand to Cassandra etc.. and give the location of that data to the >>>>>> UDF, and the UDF will, as it is called, create the map/reduce jobs and >>>>>> execute. >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> >>>>>> On Tue, Jul 1, 2014 at 9:18 AM, Srinath Perera <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> +1 we wanted to explore that more. >>>>>>> >>>>>>> However, It is not a simple UDF as this is a stateful op where we >>>>>>> feed lot of data and start a separate map reduce process. Anjana, do you >>>>>>> have any thoughts on how it can be done? >>>>>>> >>>>>>> >>>>>>> On Tue, Jul 1, 2014 at 5:37 AM, Anjana Fernando <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm just wondering if we have any way to integrate this to Hive >>>>>>>> itself (UDF?), to get results of an ML algorithm run, to a result >>>>>>>> there. A >>>>>>>> similar scenario is possible in Shark/MLlib integration. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Jun 30, 2014 at 12:28 PM, Supun Sethunga <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Im working on the $subject, and the objective is to apply Machine >>>>>>>>> Learning algorithms on the data stored by WSO2 BAM. Apache Mahout >>>>>>>>> will be >>>>>>>>> used as the ML tool, for this purpose. >>>>>>>>> >>>>>>>>> As per the discussion I had With Srinath, the procedure for >>>>>>>>> $subject would be: >>>>>>>>> >>>>>>>>> - Test a Machine Learning algorithm using Mahout libraries >>>>>>>>> within Java. >>>>>>>>> - Implement a RESTful service which provides the above >>>>>>>>> functionality. >>>>>>>>> - Since Mahout also uses Hadoop, the above service can send >>>>>>>>> Map Reduce Jobs to the Hadoop built inside the BAM. >>>>>>>>> - Deploy the service as a Carbon Component on WSO2 BAM. >>>>>>>>> >>>>>>>>> The first step is completed for now. >>>>>>>>> Any feedback is highly appreciated. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Supun >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Supun Sethunga* >>>>>>>>> Software Engineer >>>>>>>>> WSO2, Inc. >>>>>>>>> lean | enterprise | middleware >>>>>>>>> Mobile : +94 716546324 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Senior Technical Lead >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ============================ >>>>>>> Director, Research, WSO2 Inc. >>>>>>> Visiting Faculty, University of Moratuwa >>>>>>> Member, Apache Software Foundation >>>>>>> Research Scientist, Lanka Software Foundation >>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>> Phone: 0772360902 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Senior Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> ============================ >>>>> Director, Research, WSO2 Inc. >>>>> Visiting Faculty, University of Moratuwa >>>>> Member, Apache Software Foundation >>>>> Research Scientist, Lanka Software Foundation >>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>> Site: http://people.apache.org/~hemapani/ >>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>> Phone: 0772360902 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> ============================ >>> Director, Research, WSO2 Inc. >>> Visiting Faculty, University of Moratuwa >>> Member, Apache Software Foundation >>> Research Scientist, Lanka Software Foundation >>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>> Site: http://people.apache.org/~hemapani/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> lean | enterprise | middleware >> Mobile : +94 716546324 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Rajith Siriwardana* > Software Engineer > WSO2 Inc. ; http://wso2.com > *lean. enterprise. middleware* > > ------------------------------------------------------ > *http://people.apache.org/~siriwardana > <http://people.apache.org/~siriwardana>* > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- ============================ Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
