Hi Srinath, This has been reviewed with Anjana by the time we were finalizing BAM 2.4.0 M1. Shall I arrange a review with you?
Thanks, Malith On Wed, Aug 14, 2013 at 2:33 PM, Srinath Perera <[email protected]> wrote: > Have we reviewed this? > > > On Wed, Aug 14, 2013 at 1:14 PM, Malith Dhanushka <[email protected]> wrote: > >> Hi Maninda, >> >> Of course, we can add them as set of built-in analyzers and it will >> improve the scope of Hive scripts. >> >> Thanks, >> Malith >> >> >> On Wed, Aug 14, 2013 at 11:55 AM, Maninda Edirisooriya >> <[email protected]>wrote: >> >>> Hi Malith, >>> >>> Nice work. Using this new feature we can implement the requirements I >>> mentioned in the Architecture mail titled "Making BAM more useful platform >>> with well defined Class Analyzers / UDFs". WDYT? >>> >>> * >>> Maninda Edirisooriya* >>> Software Engineer >>> *WSO2, Inc. >>> *lean.enterprise.middleware. >>> >>> *Blog* : http://maninda.blogspot.com/ >>> *Phone* : +94 777603226 >>> >>> >>> On Tue, Aug 13, 2013 at 8:24 PM, Malith Dhanushka <[email protected]>wrote: >>> >>>> Hi all, >>>> >>>> I have modified the implementation according to above description and >>>> following is the modified version, >>>> >>>> - analyzer-config.xml contains the mapping details. >>>> >>>> ex : - >>>> >>>> analyzer-config.xml >>>> >>>> *<analyzerConfig xmlns="http://wso2.org/carbon/analytics";>* >>>> * <analyzers>* >>>> * <analyzer>* >>>> * <name>foo</name>* >>>> * >>>> <class>org.wso2.carbon.analytics.hive.extension.builtin.FooAnalyzer</class> >>>> * >>>> * <parameters>bar,bat1,*</parameters>* >>>> * </analyzer>* >>>> * </analyzers>* >>>> *</analyzerConfig>* >>>> >>>> parameter description, >>>> >>>> *name *- alias name which maps to the class analyzer >>>> >>>> *class* - class analyzer >>>> >>>> *parameters* - parameters that are accepted by class analyzer >>>> >>>> - This can be utilized in hive script as follows, >>>> >>>> syntax >>>> >>>> *analyzer foo(bar="value",bar1="value1",*)*; >>>> >>>> Currently there is one built-in analyzer which is resolvePath analyzer >>>> and more will be added by considering other common use cases. >>>> >>>> Thanks, >>>> Malith >>>> >>>> >>>> On Mon, Aug 5, 2013 at 11:36 AM, Anjana Fernando <[email protected]>wrote: >>>> >>>>> On Mon, Aug 5, 2013 at 11:30 AM, Maninda Edirisooriya < >>>>> [email protected]> wrote: >>>>> >>>>>> On Mon, Aug 5, 2013 at 7:58 AM, Malith Dhanushka <[email protected]>wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Implementation of the above suggested approach is in the final >>>>>>> stage. But I had a minor clarification of the implementation with >>>>>>> Anjana. >>>>>>> There we came across following drawbacks of the implemented approach, >>>>>>> >>>>>>> - Annotation should not be coupled with a class analyzer, rather it >>>>>>> should be a run time property injector to hive scripts. >>>>>>> >>>>>> Then won't the annotation feature will be bounded to Hive language? >>>>>> If so we will not be able to integrate more languages to annotations in >>>>>> future. >>>>>> >>>>> >>>>> It won't be .. the current implementation happens to only support Hive >>>>> at the moment, since that's what we have now. Not coupling with class >>>>> analyzer itself is a sign of that, because a class analyzer is anyway a >>>>> Hive functionality we have. >>>>> >>>>> Cheers, >>>>> Anjana. >>>>> >>>>> >>>>>> >>>>>>> - Abstract Annotation class adds unnecessary complications for user >>>>>>> to write a custom annotation. >>>>>>> >>>>>>> So by considering above information I am going to modify the current >>>>>>> implementation by adhering to those factors. If there any other concerns >>>>>>> and suggestions please feel free to add. >>>>>>> >>>>>>> Thanks, >>>>>>> Malith >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 10, 2013 at 1:26 PM, Malith Dhanushka >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> Hi Maninda, >>>>>>>> >>>>>>>> On Wed, Jul 10, 2013 at 11:37 AM, Maninda Edirisooriya < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> This is nice. Can we use these annotations to unify the languages, >>>>>>>>> Hive and Siddhi? If we can use this annotation framework as a >>>>>>>>> platform to >>>>>>>>> support different languages it will be very useful when it comes to >>>>>>>>> integrating other Hadoop related languages like Mahout and Pig. That >>>>>>>>> means >>>>>>>>> we can separate each language relates script using annotations. This >>>>>>>>> will >>>>>>>>> solve the problem of unifying all the languages into a single >>>>>>>>> language. >>>>>>>>> >>>>>>>> >>>>>>>> Interesting suggestion and yes ,this is gettable via annotations. >>>>>>>> But the only limitation is that the current script implementation is >>>>>>>> only >>>>>>>> available for hive. So in order to achieve language unification via >>>>>>>> annotations firstly we need to have a unified script implementation for >>>>>>>> each underlying engine (ie- Siddhi, Mahout, pig). >>>>>>>> >>>>>>>>> >>>>>>>>> And also using this annotation framework we can create a generic >>>>>>>>> Process Flow model on the data. For example we can execute several >>>>>>>>> Hive >>>>>>>>> scripts in parallel using a annotation block. And a barrier can be >>>>>>>>> introduced if all the parallel scripts should be finished before we >>>>>>>>> move >>>>>>>>> onto the next script and so on. >>>>>>>>> >>>>>>>> >>>>>>>> Yes, this can be added as a built in annotation. >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Other than that we can provide a default set of class analysers as >>>>>>>>> we have discussed in a previous mail. The value of annotations is >>>>>>>>> that we >>>>>>>>> can provide the available set of class analysers out of the box. Any >>>>>>>>> idea >>>>>>>>> about the syntax? >>>>>>>>> >>>>>>>> >>>>>>>> Each annotation is associated with a particular class analyzer, >>>>>>>> which process the given parameters via the annotation. So we can wrap >>>>>>>> that >>>>>>>> default set of class analyzers and expose them as set of built in >>>>>>>> annotations and can stick to the same syntax as previous, >>>>>>>> >>>>>>>> @script.foo(bar="value", bar1="value1",*) >>>>>>>> >>>>>>>> >>>>>>>>> * >>>>>>>>> Maninda Edirisooriya* >>>>>>>>> Software Engineer >>>>>>>>> *WSO2, Inc. >>>>>>>>> *lean.enterprise.middleware. >>>>>>>>> >>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>> *Phone* : +94 777603226 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jul 9, 2013 at 11:23 AM, Malith Dhanushka <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I have started implementing the $Subject. The idea of having an >>>>>>>>>> annotation facility is to carryout some pre-processing of Hive >>>>>>>>>> queries >>>>>>>>>> before they are being passed to the Hive engine. Currently we >>>>>>>>>> already have >>>>>>>>>> a "class analyzer" which can be used execute some custom logic as a >>>>>>>>>> part of >>>>>>>>>> a Hive script. But the main use case of annotations is to inject >>>>>>>>>> run-time >>>>>>>>>> properties to Hive execution context before the actual queries are >>>>>>>>>> carried >>>>>>>>>> out by Hive. The annotation facility would be building upon this by >>>>>>>>>> having >>>>>>>>>> set of such common analyzers which can manipulate the Hive queries >>>>>>>>>> or Hive >>>>>>>>>> execution context which it is passed to Hive query engine. >>>>>>>>>> >>>>>>>>>> Annotation Syntax, >>>>>>>>>> >>>>>>>>>> *@script.foo(bar="value", bar1="value1",*)* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Annotation scheme will be externalized by giving *abstract >>>>>>>>>> implementation of annotation* and *annotation-config.xml* file >>>>>>>>>> to provide the annotation configuration which allows third party >>>>>>>>>> annotations to be included to the system. >>>>>>>>>> >>>>>>>>>> *annotation-config.xml* >>>>>>>>>> >>>>>>>>>> <annotation> >>>>>>>>>> <name>foo</name> >>>>>>>>>> >>>>>>>>>> <class>org.wso2.carbon.analytics.hive.extension.annotation.foo</class> >>>>>>>>>> <analyzer>org.wso2.carbon.analytics.hive.extension.foo</analyzer> >>>>>>>>>> </annotation> >>>>>>>>>> >>>>>>>>>> <annotation> >>>>>>>>>> ................................ >>>>>>>>>> </annotation> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Potential use case for this in incremental data processing where >>>>>>>>>> any query associated with "*@script.incremental(foo="value1", >>>>>>>>>> bar="value2",*)*" would flag and setup the properties those are >>>>>>>>>> required to present in order for that particular query to be >>>>>>>>>> executed in an >>>>>>>>>> incremental manner.There can be many other useful additions as well. >>>>>>>>>> >>>>>>>>>> Any suggestions, thoughts are welcome. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Malith Dhanushka >>>>>>>>>> >>>>>>>>>> Engineer - Data Technologies >>>>>>>>>> *WSO2, Inc. : wso2.com* >>>>>>>>>> >>>>>>>>>> *Mobile* : +94 716 506 693 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Malith Dhanushka >>>>>>>> >>>>>>>> Engineer - Data Technologies >>>>>>>> *WSO2, Inc. : wso2.com* >>>>>>>> >>>>>>>> *Mobile* : +94 716 506 693 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Malith Dhanushka >>>>>>> >>>>>>> Engineer - Data Technologies >>>>>>> *WSO2, Inc. : wso2.com* >>>>>>> >>>>>>> *Mobile* : +94 716 506 693 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Anjana Fernando* >>>>> Technical Lead >>>>> WSO2 Inc. | http://wso2.com >>>>> lean . enterprise . middleware >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Malith Dhanushka >>>> >>>> Engineer - Data Technologies >>>> *WSO2, Inc. : wso2.com* >>>> >>>> *Mobile* : +94 716 506 693 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Malith Dhanushka >> >> Engineer - Data Technologies >> *WSO2, Inc. : wso2.com* >> >> *Mobile* : +94 716 506 693 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > ============================ > Srinath Perera, Ph.D. > http://people.apache.org/~hemapani/ > http://srinathsview.blogspot.com/ > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malith Dhanushka Engineer - Data Technologies *WSO2, Inc. : wso2.com* *Mobile* : +94 716 506 693
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
