I have a nightly Sqoop job that runs off a coordinator and uses a combination
of output datasets and EL functions to provide the workflow parameters. Nice
thing about that is you can 'backfill' with the coordinator by putting past
days in the start/end attributes of the coordinator.
Here's a piece of my workflow, using a sqoop action:
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<arg>import</arg>
<arg>--table</arg>
<arg>MyTable</arg>
<arg>--target-dir</arg>
<arg>${InputPath}</arg>
<arg>--where</arg>
<arg>"createddate between '${SDateMDY}' and '${SDateMDY}
23:59:59.997'"</arg>
</sqoop>
You can see I use two workflow properties, one for the path and one for the
date format I want.
Then the coordinator:
<coordinator-app name="NightlySqoop_coord"
frequency="${coord:days(1)}"
start="${coord_start}" end="${coord_end}" timezone="America/Los_Angeles"
xmlns="uri:oozie:coordinator:0.1">
<datasets>
<dataset name="ExportDaily" frequency="${coord:days(1)}"
initial-instance="2013-01-01T11:00Z" timezone="America/Los_Angeles">
<uri-template>${nameNode}/ExportDir/Export${YEAR}${MONTH}${DAY}</uri-template>
</dataset>
</datasets>
<output-events>
<data-out name="ExportPath" dataset="ExportDaily">
<instance>${coord:current(-1)}</instance>
</data-out>
</output-events>
<action>
<workflow>
...
<configuration>
<property>
<name>InputPath</name>
<value>${coord:dataOut('ExportPath')}</value>
</property>
<property>
<name>SDateMDY</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), -1,
'DAY'), 'MM/dd/yyyy')}</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
The -1 offset for the properties are because the job triggers every morning at
3am local time but I want it to pull the previous day's data. The output
dataset instance provides the path I need based on the uri-template, and the EL
functions coord:formatTime(), coord:dateOffset() and cood:nominalTime() build
the formatted date string.
Hope that helps.
Paul
-----Original Message-----
From: Artem Ervits [mailto:[email protected]]
Sent: Tuesday, March 05, 2013 1:49 PM
To: [email protected]
Subject: create a dir with timestamp for sqoop import
Hello all,
I am trying to do the following in Oozie workflow:
sqoop job --exec jobName -- -target-dir /test/$(date +%Y%m%d%H%M%S)
I can think of a couple of options, one is to create a sqoop action but passing
$(date +%Y%m%d%H%M%S) as argument does not work. The other option I tried was
create a shell action TIMESTAMP=$(date +%Y%m%d%H%M%S) and then in sqoop
outputdir do this: ${outputDir}/$TIMESTAMP. Unfortunately that did not work
either. Another option I tried was ${outputDir}/${timestamp()} but that format
is not recognized by the OS. Is there a replace method for ${timestamp()}
function? That would be the easiest implementation then.
--------------------
This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged. If
you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not the
named recipient, please notify us immediately by contacting the sender at the
electronic mail address noted above, and delete and destroy all copies of this
message. Thank you.
--------------------
This electronic message is intended to be for the use only of the named
recipient, and may contain information that is confidential or privileged. If
you are not the intended recipient, you are hereby notified that any
disclosure, copying, distribution or use of the contents of this message is
strictly prohibited. If you have received this message in error or are not the
named recipient, please notify us immediately by contacting the sender at the
electronic mail address noted above, and delete and destroy all copies of this
message. Thank you.
