Thanks Robert and Richard. Well I have to run this action only once a day so i'm presuming the extra work associated with mapreduce isn't going to be too much.
Having said that, thanks to richard for blog posts which will definitely be a part of solutions that I have to implement for other projects. On Tue, Nov 12, 2013 at 11:57 AM, Robert Kanter <[email protected]>wrote: > Hi, > > There are a number of built-in EL functions, including some that deal with > dates. See > > http://oozie.apache.org/docs/4.0.0/WorkflowFunctionalSpec.html#a4.2_Expression_Language_Functions > > If there isn’t an EL function that works for you, you have two options: > 1) Create your own EL function. Oozie supports loading custom EL > functions, so you don’t have to recompile Oozie, and its actually pretty > straightforward (no special classes to implement or anything like that). I > have a blog post here that goes through an example: > > http://blog.cloudera.com/blog/2013/09/how-to-write-an-el-function-in-apache-oozie/ > 2) Use the Java or Shell action to output key-value pairs as Richard > suggested. For Oozie to pick those up, you have to specify > <capture-output/> > in the action. You can then use the wf:actionData EL function to reference > those key-value pairs. I also have a blog post going through an example of > this for Shell and Java actions; its probably more involved than what you > need but it should be useful: > > http://blog.cloudera.com/blog/2013/03/how-to-use-oozie-shell-and-java-actions/ > One thing to keep in mind is that if you use the Java/Shell actions, Oozie > will run them just like the other actions by submitting an MR job to > execute them. The EL function is executed in the Oozie server, so its less > “heavy”. > > And as Richard mentioned, you can use the <param> tag in the Hive action to > pass parameters to Hive. In your Hive script, you’d use ${var_name}. Take > a look at > http://oozie.apache.org/docs/4.0.0/DG_HiveActionExtension.htmlfor > more info. > > Oh, and if you do create an EL function and think it might be helpful for > others, please consider contributing it back by creating a JIRA and posting > a patch! https://issues.apache.org/jira/browse/OOZIE > > - Robert > > > On Mon, Nov 11, 2013 at 5:31 PM, Richard Ross <[email protected] > >wrote: > > > Hello: > > > > I am by no means an expert, but we have been solving a similar problem, > > capturing parameters from a shell action (not an ssh action, as ssh > action > > is more limited in what you can do (i.e., ssh only, as I understand)), > and > > passing it through a hive action to hive scripts. The shell script just > > needs to echo the parameter as a "key=value" pair, like this: > > > > # get today and yesterday's dates however you see fit, and echo them > > echo "date_today=$date_today" > > echo "date_yesterday=$date_yesterday" > > > > Then the hive action can pass the parameters to the hive script like > this: > > > > > > > <param>DATE_TODAY=${wf:actionData('name_of_the_shell_action_goes_here')['date_today']</param> > > > > > <param>DATE_YESTERDAY=${wf:actionData('name_of_the_shell_action_goes_here')['date_yesterday']</param> > > > > The hive query in the hive action can then access the parameters like > this: > > > > SELECT * FROM foo where date = ${DATE_TODAY} > > SELECT * FROM foo where date = ${DATE_YESTERDAY} > > > > As for the specifics of the hive queries, I think that is a hive specific > > question, and not much to do with oozie. > > > > I hope this helps, > > Richard. > > > > On Nov 11, 2013, at 4:05 PMEST, P lva wrote: > > > > > I need to periodically load files from local file system into hdfs and > > > update hive table partitions. The query for updating the partitions > > depends > > > on the timestamps(select * where date = ""). > > > > > > > > > The first part I'm presuming can be done using ssh action and hdfs > -put, > > > but I dont understand how to update partitions based on date. > > > > > > I could use Oozie EL but I need today's and yesterday's date. Moreover > I > > > don't understand how to capture date and pass it as a parameter into > hive > > > script. > > > > > > How do I Do this, any ideas ? > > > > >
