Have we reviewed this?
On Wed, Aug 14, 2013 at 1:14 PM, Malith Dhanushka <[email protected]> wrote: > Hi Maninda, > > Of course, we can add them as set of built-in analyzers and it will > improve the scope of Hive scripts. > > Thanks, > Malith > > > On Wed, Aug 14, 2013 at 11:55 AM, Maninda Edirisooriya > <[email protected]>wrote: > >> Hi Malith, >> >> Nice work. Using this new feature we can implement the requirements I >> mentioned in the Architecture mail titled "Making BAM more useful platform >> with well defined Class Analyzers / UDFs". WDYT? >> >> * >> Maninda Edirisooriya* >> Software Engineer >> *WSO2, Inc. >> *lean.enterprise.middleware. >> >> *Blog* : http://maninda.blogspot.com/ >> *Phone* : +94 777603226 >> >> >> On Tue, Aug 13, 2013 at 8:24 PM, Malith Dhanushka <[email protected]>wrote: >> >>> Hi all, >>> >>> I have modified the implementation according to above description and >>> following is the modified version, >>> >>> - analyzer-config.xml contains the mapping details. >>> >>> ex : - >>> >>> analyzer-config.xml >>> >>> *<analyzerConfig xmlns="http://wso2.org/carbon/analytics";>* >>> * <analyzers>* >>> * <analyzer>* >>> * <name>foo</name>* >>> * >>> <class>org.wso2.carbon.analytics.hive.extension.builtin.FooAnalyzer</class> >>> * >>> * <parameters>bar,bat1,*</parameters>* >>> * </analyzer>* >>> * </analyzers>* >>> *</analyzerConfig>* >>> >>> parameter description, >>> >>> *name *- alias name which maps to the class analyzer >>> >>> *class* - class analyzer >>> >>> *parameters* - parameters that are accepted by class analyzer >>> >>> - This can be utilized in hive script as follows, >>> >>> syntax >>> >>> *analyzer foo(bar="value",bar1="value1",*)*; >>> >>> Currently there is one built-in analyzer which is resolvePath analyzer >>> and more will be added by considering other common use cases. >>> >>> Thanks, >>> Malith >>> >>> >>> On Mon, Aug 5, 2013 at 11:36 AM, Anjana Fernando <[email protected]>wrote: >>> >>>> On Mon, Aug 5, 2013 at 11:30 AM, Maninda Edirisooriya <[email protected] >>>> > wrote: >>>> >>>>> On Mon, Aug 5, 2013 at 7:58 AM, Malith Dhanushka <[email protected]>wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Implementation of the above suggested approach is in the final stage. >>>>>> But I had a minor clarification of the implementation with Anjana. There >>>>>> we >>>>>> came across following drawbacks of the implemented approach, >>>>>> >>>>>> - Annotation should not be coupled with a class analyzer, rather it >>>>>> should be a run time property injector to hive scripts. >>>>>> >>>>> Then won't the annotation feature will be bounded to Hive language? If >>>>> so we will not be able to integrate more languages to annotations in >>>>> future. >>>>> >>>> >>>> It won't be .. the current implementation happens to only support Hive >>>> at the moment, since that's what we have now. Not coupling with class >>>> analyzer itself is a sign of that, because a class analyzer is anyway a >>>> Hive functionality we have. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> >>>>> >>>>>> - Abstract Annotation class adds unnecessary complications for user >>>>>> to write a custom annotation. >>>>>> >>>>>> So by considering above information I am going to modify the current >>>>>> implementation by adhering to those factors. If there any other concerns >>>>>> and suggestions please feel free to add. >>>>>> >>>>>> Thanks, >>>>>> Malith >>>>>> >>>>>> >>>>>> On Wed, Jul 10, 2013 at 1:26 PM, Malith Dhanushka <[email protected]>wrote: >>>>>> >>>>>>> Hi Maninda, >>>>>>> >>>>>>> On Wed, Jul 10, 2013 at 11:37 AM, Maninda Edirisooriya < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> This is nice. Can we use these annotations to unify the languages, >>>>>>>> Hive and Siddhi? If we can use this annotation framework as a platform >>>>>>>> to >>>>>>>> support different languages it will be very useful when it comes to >>>>>>>> integrating other Hadoop related languages like Mahout and Pig. That >>>>>>>> means >>>>>>>> we can separate each language relates script using annotations. This >>>>>>>> will >>>>>>>> solve the problem of unifying all the languages into a single language. >>>>>>>> >>>>>>> >>>>>>> Interesting suggestion and yes ,this is gettable via annotations. >>>>>>> But the only limitation is that the current script implementation is >>>>>>> only >>>>>>> available for hive. So in order to achieve language unification via >>>>>>> annotations firstly we need to have a unified script implementation for >>>>>>> each underlying engine (ie- Siddhi, Mahout, pig). >>>>>>> >>>>>>>> >>>>>>>> And also using this annotation framework we can create a generic >>>>>>>> Process Flow model on the data. For example we can execute several Hive >>>>>>>> scripts in parallel using a annotation block. And a barrier can be >>>>>>>> introduced if all the parallel scripts should be finished before we >>>>>>>> move >>>>>>>> onto the next script and so on. >>>>>>>> >>>>>>> >>>>>>> Yes, this can be added as a built in annotation. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Other than that we can provide a default set of class analysers as >>>>>>>> we have discussed in a previous mail. The value of annotations is that >>>>>>>> we >>>>>>>> can provide the available set of class analysers out of the box. Any >>>>>>>> idea >>>>>>>> about the syntax? >>>>>>>> >>>>>>> >>>>>>> Each annotation is associated with a particular class analyzer, >>>>>>> which process the given parameters via the annotation. So we can wrap >>>>>>> that >>>>>>> default set of class analyzers and expose them as set of built in >>>>>>> annotations and can stick to the same syntax as previous, >>>>>>> >>>>>>> @script.foo(bar="value", bar1="value1",*) >>>>>>> >>>>>>> >>>>>>>> * >>>>>>>> Maninda Edirisooriya* >>>>>>>> Software Engineer >>>>>>>> *WSO2, Inc. >>>>>>>> *lean.enterprise.middleware. >>>>>>>> >>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>> *Phone* : +94 777603226 >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jul 9, 2013 at 11:23 AM, Malith Dhanushka >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I have started implementing the $Subject. The idea of having an >>>>>>>>> annotation facility is to carryout some pre-processing of Hive queries >>>>>>>>> before they are being passed to the Hive engine. Currently we already >>>>>>>>> have >>>>>>>>> a "class analyzer" which can be used execute some custom logic as a >>>>>>>>> part of >>>>>>>>> a Hive script. But the main use case of annotations is to inject >>>>>>>>> run-time >>>>>>>>> properties to Hive execution context before the actual queries are >>>>>>>>> carried >>>>>>>>> out by Hive. The annotation facility would be building upon this by >>>>>>>>> having >>>>>>>>> set of such common analyzers which can manipulate the Hive queries or >>>>>>>>> Hive >>>>>>>>> execution context which it is passed to Hive query engine. >>>>>>>>> >>>>>>>>> Annotation Syntax, >>>>>>>>> >>>>>>>>> *@script.foo(bar="value", bar1="value1",*)* >>>>>>>>> >>>>>>>>> >>>>>>>>> Annotation scheme will be externalized by giving *abstract >>>>>>>>> implementation of annotation* and *annotation-config.xml* file to >>>>>>>>> provide the annotation configuration which allows third party >>>>>>>>> annotations >>>>>>>>> to be included to the system. >>>>>>>>> >>>>>>>>> *annotation-config.xml* >>>>>>>>> >>>>>>>>> <annotation> >>>>>>>>> <name>foo</name> >>>>>>>>> >>>>>>>>> <class>org.wso2.carbon.analytics.hive.extension.annotation.foo</class> >>>>>>>>> <analyzer>org.wso2.carbon.analytics.hive.extension.foo</analyzer> >>>>>>>>> </annotation> >>>>>>>>> >>>>>>>>> <annotation> >>>>>>>>> ................................ >>>>>>>>> </annotation> >>>>>>>>> >>>>>>>>> >>>>>>>>> Potential use case for this in incremental data processing where >>>>>>>>> any query associated with "*@script.incremental(foo="value1", >>>>>>>>> bar="value2",*)*" would flag and setup the properties those are >>>>>>>>> required to present in order for that particular query to be executed >>>>>>>>> in an >>>>>>>>> incremental manner.There can be many other useful additions as well. >>>>>>>>> >>>>>>>>> Any suggestions, thoughts are welcome. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Malith Dhanushka >>>>>>>>> >>>>>>>>> Engineer - Data Technologies >>>>>>>>> *WSO2, Inc. : wso2.com* >>>>>>>>> >>>>>>>>> *Mobile* : +94 716 506 693 >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Malith Dhanushka >>>>>>> >>>>>>> Engineer - Data Technologies >>>>>>> *WSO2, Inc. : wso2.com* >>>>>>> >>>>>>> *Mobile* : +94 716 506 693 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Malith Dhanushka >>>>>> >>>>>> Engineer - Data Technologies >>>>>> *WSO2, Inc. : wso2.com* >>>>>> >>>>>> *Mobile* : +94 716 506 693 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Malith Dhanushka >>> >>> Engineer - Data Technologies >>> *WSO2, Inc. : wso2.com* >>> >>> *Mobile* : +94 716 506 693 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Malith Dhanushka > > Engineer - Data Technologies > *WSO2, Inc. : wso2.com* > > *Mobile* : +94 716 506 693 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- ============================ Srinath Perera, Ph.D. http://people.apache.org/~hemapani/ http://srinathsview.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
