Hi All, Currently to implement event based notification, Oozie supports integration with HCatalog to receive notification based on partitions availability. Now, this works fine if the coordinator knows what specific partition to wait for. Consider the below use case. System A generates a report daily that aggregates customer records for that day. We want to schedule a workflow once the report is generated. Normally we would expect report to be generated once a day and hence the implicit frequency for workflow would be 1 day. However, on some days, there could be multiple re-runs of the report generation process. This means there are now multiple versions of that day's report. To trigger the workflow in Oozie, we could have a HCAT partition specification such as : Date/Run_Type (e.g. 01-Jan-2015/ReportGeneration) In coordinator we would have following setup as input.. <uri-template>hcat://myhcatmetastore:9080/database1/table1/Date=01-Jan-2015;Run_Type=ReportGeneration</uri-template>
Now, the above URI does not mention anything about version. What we would want to do is create partitions such as follows * 01-Jan-2015;Run_Type=ReportGeneration;Version=1 * 01-Jan-2015;Run_Type=ReportGeneration;Version=2 * 02-Jan-2015;Run_Type=ReportGeneration;Version=1 * 03-Jan-2015;Run_Type=ReportGeneration;Version=1 And want to trigger Workflow for each of the above partitions. Since coordinator does not know what specific version to wait for, how to configure URI template for this ? <uri-template>hcat://myhcatmetastore:9080/database1/table1/Date=01-Jan-2015;Run_Type=ReportGeneration;Version=???</uri-template> Currently the work around is to have coordinator resort to polling, but we want an event-based implementation for this. One option could be to define a custom EL function that gets the Version information, as below. (haven't tested it yet) <uri-template>hcat://myhcatmetastore:9080/database1/table1/Date=01-Jan-2015;Run_Type=ReportGeneration;Version=Get_Version()</uri-template> Please let us know if you think there is a better way to achieve this. Regards Vishal Joshi _______________________________________________ This message is for information purposes only, it is not a recommendation, advice, offer or solicitation to buy or sell a product or service nor an official confirmation of any transaction. It is directed at persons who are professionals and is not intended for retail customer use. Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer. For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding market commentary from Barclays Sales and/or Trading, who are active market participants; and in respect of Barclays Research, including disclosures relating to specific issuers, please see http://publicresearch.barclays.com. _______________________________________________
