I'm creating a coordinator application. See the attachment for workflow and
coordinator xml files.
On Thu, Jun 25, 2015 at 9:37 AM, Oussama Chougna <[email protected]>
wrote:
> Vincent,
> Are you creating a workflow ony oozie application or a coordinator
> application?
> Could you post some of your xml definition files?
> Thanks.
>
> Oussama Chougna
>
> > Date: Thu, 25 Jun 2015 08:27:35 -0400
> > Subject: Re: HDFS EL Function - fs:dirSize(String path)
> > From: [email protected]
> > To: [email protected]
> >
> > Thanks for the suggestion Oussama, unfortunately it did not resolve my
> > issue :(
> >
> > <param>byt_wrt=${fs:dirSize('/user/peppy/testing/p_dt=coord:
> > conf('pDate')')}</param>
> >
> > Resulted in the following:
> >
> > Error Code : EL_ERROR
> > Error Message : Encountered "pDate", expected one of [".", ">", "gt",
> > "<", "lt", "==", "eq", "<=", "le", ">=", "ge", "!=", "ne", ")", ",", "[",
> > "+", "-", "*", "/", "div", "%", "mod", "and", "&&", "or", "||", "?"]
> >
> > On Thu, Jun 25, 2015 at 3:13 AM, Oussama Chougna <
> [email protected]>
> > wrote:
> >
> > > You should try this:
> > > If you are using a coordinator:
> > >
> > >
> <param>byt_wrt=${fs:dirSize('/user/peppy/testing/p_dt=coord:conf('pDate')')}</param>
> > > If you only using a workflow:
> > >
> > >
> <param>byt_wrt=${fs:dirSize('/user/peppy/testing/p_dt=wf:conf('pDate')')}</param>
> > >
> > > Nested EL does not work.
> > >
> > >
> > > Good luck,
> > > Cheers.
> > > Oussama Chougna
> > >
> > >
> > >
> > > > Date: Wed, 24 Jun 2015 20:49:32 +0000
> > > > From: [email protected]
> > > > To: [email protected]
> > > > Subject: Re: HDFS EL Function - fs:dirSize(String path)
> > > >
> > > >
> <param>byt_wrt=${fs:dirSize('/user/peppy/testing/p_dt=pDate')}</param> .
> > > pDate without ${}.
> > > > From: Vincent Peplinski <[email protected]>
> > > > To: [email protected]
> > > > Sent: Wednesday, June 24, 2015 12:41 PM
> > > > Subject: HDFS EL Function - fs:dirSize(String path)
> > > >
> > > > Hello,
> > > >
> > > > I'm trying to use the fs:dirSize() HDFS EL function in one of my
> > > workflows
> > > > action nodes but I'm having some difficulty and I hope someone can
> help.
> > > >
> > > > When I call the function using a literal string such as:
> > > >
> > > >
> <param>byt_wrt=${fs:dirSize('/user/peppy/testing/p_dt=20150624')}</param>
> > > >
> > > >
> > > > The function returns the size of the file as expected. However; when
> I
> > > try
> > > > the following:
> > > >
> > > >
> <param>byt_wrt=${fs:dirSize('/user/peppy/testing/p_dt=${pDate}')}</param>
> > > >
> > > > The workflow fails with an EL_ERROR - Illegal character in path
> index 26
> > > > /user/peppy/testing/p_dt=${pDate}.
> > > >
> > > >
> > > > How can I pass a variable to the function that will be properly
> expanded?
> > > >
> > > > Thank you in advance.
> > > >
> > > > V.
> > > >
> > > >
> > > >
> > >
> > >
>
>
<coordinator-app name="stats_test-${dcNumber}:region=${forRegion}"
frequency="${coord:days(1)}" start="${start}" end="${end}" timezone="UTC"
xmlns="uri:oozie:coordinator:0.4">
<controls>
<timeout>${timeOut}</timeout>
<concurrency>${concurrency}</concurrency>
<execution>${exeOrder}</execution>
<throttle>${throttle}</throttle>
</controls>
<datasets>
<dataset name="din-1" frequency="${coord:days(1)}"
initial-instance="${initInstance}" timezone="UTC">
<uri-template>${inputDir1}/p_dt=${YEAR}${MONTH}${DAY}</uri-template>
<done-flag>_READY</done-flag>
</dataset>
</datasets>
<input-events>
<data-in name="data1" dataset="din-1">
<instance>${coord:current(0)}</instance>
</data-in>
</input-events>
<action>
<workflow>
<app-path>${workflowAppUri}</app-path>
<configuration>
<property>
<name>jobTracker</name>
<value>${jobTracker}</value>
</property>
<property>
<name>nameNode</name>
<value>${nameNode}</value>
</property>
<property>
<name>queueName</name>
<value>${queueName}</value>
</property>
<property>
<name>pDate</name>
<value>${coord:formatTime(coord:dateOffset(coord:nominalTime(), 0, 'DAY'),
'yyyyMMdd')}</value>
</property>
</configuration>
</workflow>
</action>
</coordinator-app>
<!-- -->
<workflow-app xmlns="uri:oozie:workflow:0.4"
name="stats_test-${dcNumber}:r=${forRegion}">
<global>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/oozie/hive-conf/hive-site.xml</job-xml>
<configuration>
<property>
<name>mapreduce.job.queuename</name>
<value>${queueName}</value>
</property>
</configuration>
</global>
<credentials>
<credential name='hive_auth' type='hcat'>
<property>
<name>hcat.metastore.uri</name>
<value>${hcatURI}</value>
</property>
<property>
<name>hcat.metastore.principal</name>
<value>${hcatPrincipal}</value>
</property>
</credential>
</credentials>
<start to="Update_Stats_G_MFH"/>
<action name="Update_Stats_G_MFH" cred='hive_auth'>
<hive xmlns="uri:oozie:hive-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/oozie/hive-conf/hive-site.xml</job-xml>
<script>update_stats.hql</script>
<param>byt_wrt=${fs:dirSize('/user/hive/warehouse/bds_global.db/global_mfh_daily/p_dte=coord:conf('pDate')'}</param>
</hive>
<ok to="End"/>
<error to="Report"/>
</action>
<action name="Report">
<email xmlns="uri:oozie:email-action:0.1">
<to>${notifyList}</to>
<subject>Email notification for ${wf:name()}</subject>
<body>${wf:id()}@${wf:lastErrorNode()} Failed:
${wf:errorMessage(wf:lastErrorNode())}.</body>
</email>
<ok to="Fail"/>
<error to="Fail"/>
</action>
<kill name="Fail">
<message>Job Failed: error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="End"/>
</workflow-app>
