Thanks for the clarification. I was thinking of adopting this to improve WSO2 Cloud Monitor by introducing ML to the system. Currently we are using UDFs and UDAFs for the purpose. :)
Thanks, Rajith On Fri, Jul 25, 2014 at 9:16 AM, Srinath Perera <[email protected]> wrote: > Currently, you have to edit code the add new algorithms. We will make it > configurable eventually, but at this stage we do not need a framework IMHO. > > --Srinath > > > On Thu, Jul 24, 2014 at 7:46 PM, Rajith Siriwardena <[email protected]> > wrote: > >> >> Okay got it! :) >> >> @supun : If you don't mind can you share the architecture diagram. I'm >> bit interested in how it's done. how can we introduce new algorithms to the >> component? >> >> Thanks, >> Rajith >> >> >> On Thu, Jul 24, 2014 at 10:33 AM, Supun Sethunga <[email protected]> wrote: >> >>> Hi Rajith, >>> >>> The service is a stand-alone carbon component, and not a UDAF. Hive will >>> only be used to retrieve and store the data which will eventually used by >>> the service. >>> >>> Regards, >>> Supun >>> >>> >>> On Thu, Jul 24, 2014 at 9:28 AM, Rajith Siriwardena <[email protected]> >>> wrote: >>> >>>> Hi Supun, >>>> >>>> Are you using UDAFs for the purpose? Can explain a bit how it's done? >>>> >>>> >>>> On Wed, Jul 23, 2014 at 5:30 PM, Supun Sethunga <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> The back-end service for the $subject has been implemented. For the >>>>> time being, it supports three algorithms (Naive Bayes, Logistic Regression >>>>> and Multilayer Perceptrons). The service accepts a JSON object having the >>>>> input data file path, output path where the created the model should be >>>>> saved, the algorithm to be used for modeling (one of the above three >>>>> algos), and the hyper-parameters and their values. >>>>> >>>>> Next goal would be to develop a carbon UI component, which allows a >>>>> user to call the above service, in a form of a wizard. The wizard would >>>>> provide a step-by-step guide to the user to create a ML model for any set >>>>> of data. >>>>> >>>>> Appreciate any suggestions. >>>>> >>>>> Regards, >>>>> Supun >>>>> >>>>> >>>>> On Fri, Jul 4, 2014 at 7:39 AM, Srinath Perera <[email protected]> >>>>> wrote: >>>>> >>>>>> Anjana and myself discussed. >>>>>> >>>>>> UDFs does not work. UDFs take an event as the input while mahout >>>>>> takes a complete dataset. >>>>>> >>>>>> May be we can use UDAF (A = Aggregate), but has to explore more. >>>>>> >>>>>> >>>>>> On Tue, Jul 1, 2014 at 10:03 AM, Anjana Fernando <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I see, sure, I was thinking of doing all the operations, including >>>>>>> the training operations using an UDF. Will come and meet you. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> >>>>>>> On Tue, Jul 1, 2014 at 9:56 AM, Srinath Perera <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> No, we need to get the data, preprocess them using hive, and send >>>>>>>> all the data (not 1-2 values, rather say 10 millions values) to >>>>>>>> training >>>>>>>> phase. Lets chat f2f a bit. >>>>>>>> >>>>>>>> --Srinath >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 1, 2014 at 6:24 AM, Anjana Fernando <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I was simply thinking, the UDF could directly mapped to some basic >>>>>>>>> Mahout operation it implements, and the input/output should be given >>>>>>>>> as >>>>>>>>> parameters to the UDF, so probably, we can publish some input data >>>>>>>>> beforehand to Cassandra etc.. and give the location of that data to >>>>>>>>> the >>>>>>>>> UDF, and the UDF will, as it is called, create the map/reduce jobs and >>>>>>>>> execute. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jul 1, 2014 at 9:18 AM, Srinath Perera <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> +1 we wanted to explore that more. >>>>>>>>>> >>>>>>>>>> However, It is not a simple UDF as this is a stateful op where we >>>>>>>>>> feed lot of data and start a separate map reduce process. Anjana, do >>>>>>>>>> you >>>>>>>>>> have any thoughts on how it can be done? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 1, 2014 at 5:37 AM, Anjana Fernando <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I'm just wondering if we have any way to integrate this to Hive >>>>>>>>>>> itself (UDF?), to get results of an ML algorithm run, to a result >>>>>>>>>>> there. A >>>>>>>>>>> similar scenario is possible in Shark/MLlib integration. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Anjana. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Jun 30, 2014 at 12:28 PM, Supun Sethunga < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Im working on the $subject, and the objective is to apply >>>>>>>>>>>> Machine Learning algorithms on the data stored by WSO2 BAM. Apache >>>>>>>>>>>> Mahout >>>>>>>>>>>> will be used as the ML tool, for this purpose. >>>>>>>>>>>> >>>>>>>>>>>> As per the discussion I had With Srinath, the procedure for >>>>>>>>>>>> $subject would be: >>>>>>>>>>>> >>>>>>>>>>>> - Test a Machine Learning algorithm using Mahout libraries >>>>>>>>>>>> within Java. >>>>>>>>>>>> - Implement a RESTful service which provides the above >>>>>>>>>>>> functionality. >>>>>>>>>>>> - Since Mahout also uses Hadoop, the above service can send >>>>>>>>>>>> Map Reduce Jobs to the Hadoop built inside the BAM. >>>>>>>>>>>> - Deploy the service as a Carbon Component on WSO2 BAM. >>>>>>>>>>>> >>>>>>>>>>>> The first step is completed for now. >>>>>>>>>>>> Any feedback is highly appreciated. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Supun >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Supun Sethunga* >>>>>>>>>>>> Software Engineer >>>>>>>>>>>> WSO2, Inc. >>>>>>>>>>>> lean | enterprise | middleware >>>>>>>>>>>> Mobile : +94 716546324 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>> Senior Technical Lead >>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ============================ >>>>>>>>>> Director, Research, WSO2 Inc. >>>>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>>>> Member, Apache Software Foundation >>>>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>> Phone: 0772360902 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Senior Technical Lead >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Director, Research, WSO2 Inc. >>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>> Member, Apache Software Foundation >>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> ============================ >>>>>> Director, Research, WSO2 Inc. >>>>>> Visiting Faculty, University of Moratuwa >>>>>> Member, Apache Software Foundation >>>>>> Research Scientist, Lanka Software Foundation >>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>> Site: http://people.apache.org/~hemapani/ >>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>> Phone: 0772360902 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Rajith Siriwardana* >>>> Software Engineer >>>> WSO2 Inc. ; http://wso2.com >>>> *lean. enterprise. middleware* >>>> >>>> ------------------------------------------------------ >>>> *http://people.apache.org/~siriwardana >>>> <http://people.apache.org/~siriwardana>* >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >> >> -- >> *Rajith Siriwardana* >> Software Engineer >> WSO2 Inc. ; http://wso2.com >> *lean. enterprise. middleware* >> >> ------------------------------------------------------ >> *http://people.apache.org/~siriwardana >> <http://people.apache.org/~siriwardana>* >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > ============================ > Director, Research, WSO2 Inc. > Visiting Faculty, University of Moratuwa > Member, Apache Software Foundation > Research Scientist, Lanka Software Foundation > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Rajith Siriwardana* Software Engineer WSO2 Inc. ; http://wso2.com *lean. enterprise. middleware* ------------------------------------------------------ *http://people.apache.org/~siriwardana <http://people.apache.org/~siriwardana>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
