Anjana and myself discussed. UDFs does not work. UDFs take an event as the input while mahout takes a complete dataset.
May be we can use UDAF (A = Aggregate), but has to explore more. On Tue, Jul 1, 2014 at 10:03 AM, Anjana Fernando <[email protected]> wrote: > I see, sure, I was thinking of doing all the operations, including the > training operations using an UDF. Will come and meet you. > > Cheers, > Anjana. > > > On Tue, Jul 1, 2014 at 9:56 AM, Srinath Perera <[email protected]> wrote: > >> No, we need to get the data, preprocess them using hive, and send all the >> data (not 1-2 values, rather say 10 millions values) to training phase. >> Lets chat f2f a bit. >> >> --Srinath >> >> >> On Tue, Jul 1, 2014 at 6:24 AM, Anjana Fernando <[email protected]> wrote: >> >>> Hi, >>> >>> I was simply thinking, the UDF could directly mapped to some basic >>> Mahout operation it implements, and the input/output should be given as >>> parameters to the UDF, so probably, we can publish some input data >>> beforehand to Cassandra etc.. and give the location of that data to the >>> UDF, and the UDF will, as it is called, create the map/reduce jobs and >>> execute. >>> >>> Cheers, >>> Anjana. >>> >>> >>> On Tue, Jul 1, 2014 at 9:18 AM, Srinath Perera <[email protected]> wrote: >>> >>>> +1 we wanted to explore that more. >>>> >>>> However, It is not a simple UDF as this is a stateful op where we feed >>>> lot of data and start a separate map reduce process. Anjana, do you have >>>> any thoughts on how it can be done? >>>> >>>> >>>> On Tue, Jul 1, 2014 at 5:37 AM, Anjana Fernando <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm just wondering if we have any way to integrate this to Hive itself >>>>> (UDF?), to get results of an ML algorithm run, to a result there. A >>>>> similar >>>>> scenario is possible in Shark/MLlib integration. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> >>>>> On Mon, Jun 30, 2014 at 12:28 PM, Supun Sethunga <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Im working on the $subject, and the objective is to apply Machine >>>>>> Learning algorithms on the data stored by WSO2 BAM. Apache Mahout will be >>>>>> used as the ML tool, for this purpose. >>>>>> >>>>>> As per the discussion I had With Srinath, the procedure for $subject >>>>>> would be: >>>>>> >>>>>> - Test a Machine Learning algorithm using Mahout libraries within >>>>>> Java. >>>>>> - Implement a RESTful service which provides the above >>>>>> functionality. >>>>>> - Since Mahout also uses Hadoop, the above service can send Map >>>>>> Reduce Jobs to the Hadoop built inside the BAM. >>>>>> - Deploy the service as a Carbon Component on WSO2 BAM. >>>>>> >>>>>> The first step is completed for now. >>>>>> Any feedback is highly appreciated. >>>>>> >>>>>> Thanks, >>>>>> Supun >>>>>> >>>>>> -- >>>>>> *Supun Sethunga* >>>>>> Software Engineer >>>>>> WSO2, Inc. >>>>>> lean | enterprise | middleware >>>>>> Mobile : +94 716546324 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Senior Technical Lead >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>> >>>> >>>> >>>> -- >>>> ============================ >>>> Director, Research, WSO2 Inc. >>>> Visiting Faculty, University of Moratuwa >>>> Member, Apache Software Foundation >>>> Research Scientist, Lanka Software Foundation >>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> Phone: 0772360902 >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> ============================ >> Director, Research, WSO2 Inc. >> Visiting Faculty, University of Moratuwa >> Member, Apache Software Foundation >> Research Scientist, Lanka Software Foundation >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- ============================ Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
