On Wed, Jun 12, 2013 at 3:23 PM, Srinath Perera <[email protected]> wrote:
> +1 I think it is useful. > > We should define UDF (user define functions) that conform to Hive model as > close as possible. > Agreed. Proposed "Hive annotation scheme" is pretty much align with this requirement which i have raised before by the subject of "Path inconsistency issue in BAM Hive scripts" in Archi mailing list. This is some what slimier to the template library strategy in ESB and what we have at the end is a library of annotations which expose the common analyzers so they will be easy to utilize when scripting. We are hoping to add this feature for the next BAM release. > --Srinath > > > On Wed, Jun 12, 2013 at 1:13 PM, Maninda Edirisooriya <[email protected]>wrote: > >> The main configuration point of analytics in BAM is its Hive scripts. At >> the moment we only support Hive language. But Hive has many drawbacks. For >> example there is no generic way to execute a SQL script on a RDBMS table in >> a Hive command. We had that requirement to execute the "DROP TABLE" query >> in Hive. Though there was a work around it was not a proper solution. >> >> On the other hand AFAIK there is no proper listening mechanism for finish >> of execution of a Hive script. Such a technique is important to notify >> and/or pass data to another part of the WSO2 stack or call another part of >> the system. (e.g., invoke a proxy service in ESB after a certain Hive code >> line is executed) This kind of push modal is required for BAM when BAM is >> used for alerting scenarios. Yes, CEP is used for alerting but BAM can be >> used for not-urgent alerting. For example send an e-mail to marketing team >> after the sales has dropped to a certain level during one month. This can >> be done with BAM. Data push modal is required for reducing the latency as >> much as possible. >> >> I have come across another requirement for Class Analyzer / UDF with XML >> processing. Though simple XML can be processed with existing UDFs provided >> with Hive, XML strings with namespaces are not supported. We should add our >> own UDF for Hive level SOAP message processing. >> >> And also, already we pack a set of Class analyzers for IP to geo-location >> mapping and for deleting Cassandra rows for archiving purposes. Though they >> are packaged with BAM we do not explicitly mention about their existence in >> docs, though they are very useful for the users. >> >> I think it is the time to upgrade Hive script API we provide, with the >> new Class Analyzers / UDFs we have introduced. And also we should implement >> the Class Analyzers / UDFs for the commonly used scenarios such as the >> requirements I have mentioned earlier. This will improve the user >> experience by improving the userbility of Hive for broad range of >> solutions. And finally it will make the BAM analyzer similar to the >> Mediation Sequence in ESB which will improve the integration capability. >> >> Here are some of the Class analyzers / UDFs I think that will be useful >> for packing with BAM. >> >> 1. Executing a generic SQL statement (as described in the above example) >> 2. Web service / REST API calling (for invoking AS or ESB) >> 3. Java class execution - Similar to Jaggery Package host object. This >> should make it possible call any Java class in the OSGi environment >> 4. Thrift call execution - will help to pump data back to CEP and then >> take actions inside CEP >> 5. Registry value getting and setting - This will help a lot to share >> global values between Hive scripts while execution. >> 6. Hive script invoking - This will enable to asynchronously invoke >> another Hive script during execution of the original script. >> 7. Conditioning and iterating - will make it possible to stop/continue >> the script sequence by conditioning and will possible to execute a script >> as a for loop or while loop. >> >> There may be other useful features as well for defining the API. WDYT? >> > +1 for the list of useful Class analyzers / UDFs. And one other addition is, Hive Script resolve path - resolve the paths depending on the platform before execute the hive script. Thanks, Malith * >> Maninda Edirisooriya* >> Software Engineer >> *WSO2, Inc. >> *lean.enterprise.middleware. >> >> *Blog* : http://maninda.blogspot.com/ >> *Phone* : +94 777603226 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > ============================ > Srinath Perera, Ph.D. > http://people.apache.org/~hemapani/ > http://srinathsview.blogspot.com/ > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Malith Dhanushka Engineer - Data Technologies *WSO2, Inc. : wso2.com* lean.enterprise.middleware. *Blog* : mmalithh.blogspot.com *Linked-i*n : http://lk.linkedin.com/in/malithd *Twitter* : https://twitter.com/MalithWD *Mobile* : +94 716 506 693
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
