Thanks for your reply Oussama. However, based on what I have read It
should be possible to access configuration properties in your
workflow.xml with the help of EL.
Reference: "Apache Oozie" book from O'Reilly, Example 5-13
Possible Source code reference: substituteVars function in
https://github.com/apache/oozie/blob/release-4.0.0/core/src/main/java/org/apache/oozie/util/XConfiguration.java
If this feature is not present in Oozie, then the use of
${wf:conf("prop.name")} is very limited.
Let me describe my situation as closely as possible:
I need to copy a source file to 3 or 4 different destinations.
All the paths follow similar convention like:
<namenode>/<parentdir>/yyyy/yyyymm/yyyymmdd/<childdir>/<fileName>
I am using ${coord:formatTime(coord:nominalTime(), "yyyy")} to get the
yyyy string and similarly for mm and dd part of the path.
What I would like to do is use the coord:formatTime function as few
places as possible to make it easy to change in future. For example:
<configuration>
<property>
<name>year</name>
<value>${coord:formatTime(coord:nominalTime(), "yyyy")}</value>
</property>
<property>
<name>month</name>
<value>${coord:formatTime(coord:nominalTime(), "MM")}</value>
</property>
<property>
<name>date</name>
<value>${coord:formatTime(coord:nominalTime(), "dd")}</value>
</property>
<property>
<name>path_containing_date</name>
<value>${year}/${year}${month}/${year}${month}${date}</value>
</property>
<property>
<name>source_path</name>
<value>${namenodeA_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
<property>
<name>destination_path_1</name>
<value>${namenode1_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
<property>
<name>destination_path_2</name>
<value>${namenode2_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
<property>
<name>destination_path_3</name>
<value>${namenode3_with_parent_path}/${path_containing_date}/${remaining_path}</value>
</property>
</configuration>
This I hope would be more maintainable than using formatTime
everywhere I need date. This way If I want to run manually for a
different date, I can just pass in the year, month and date values.
If this feature is not present right now, Is there a plan to implement it?
On Mon, Oct 19, 2015 at 8:06 AM, Oussama Chougna
<[email protected]> wrote:
> Hello Sunil,
> That's not how it works. In the global section you can define oozie
> properties which are global for all actions in the workflow, like
> mapred.job.queue.name, job-xml etc..
> What you can do is put 'param1' in your job.properties, then you can resolve
> it with EL via ${param1}.
>
> Best,
>
> Oussama Chougna
>
>
>> Date: Mon, 19 Oct 2015 07:27:22 -0700
>> Subject: parameter/variable substitution not working in oozie 4.0.0-cdh5.3.2
>> From: [email protected]
>> To: [email protected]
>>
>> Hi,
>>
>> My Oozie server version: 4.0.0-cdh5.3.2
>> I have a variable A, the value of which would be a combination of
>> other variables (B, C etc.). However, oozie is unable to recognize and
>> substitute values from B, C to form value of A. It throws:
>> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
>>
>> Please let me know if it is a known issue in the version I am using.
>>
>> Here is a simple workflow example I am trying:
>> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
>> <global>
>> <configuration>
>> <property>
>> <name>param1</name>
>> <value>idHere</value>
>> </property>
>> </configuration>
>> </global>
>> <start to="test-echo-shell-1"/>
>> <action name="test-echo-shell-1">
>> <shell xmlns="uri:oozie:shell-action:0.1">
>> <job-tracker>${jobTracker}</job-tracker>
>> <name-node>${nameNode}</name-node>
>> <exec>/bin/echo</exec>
>> <argument>${param1}</argument>
>> <capture-output/>
>> </shell>
>> <ok to="end"/>
>> <error to="kill"/>
>> </action>
>> <kill name="kill">
>> <message>Action failed, error
>> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>> </kill>
>> <end name="end"/>
>> </workflow-app>
>>
>> The above example throws:
>> javax.servlet.jsp.el.ELException: variable [param1] cannot be resolved
>> at
>> org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106)
>> at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124)
>> at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
>> at
>> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274)
>> at
>> org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190)
>> at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:203)
>> at
>> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:188)
>> at
>> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
>> at org.apache.oozie.command.XCommand.call(XCommand.java:281)
>> at
>> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
>> at
>> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
>> at
>> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
>>
>>
>> Here is another example which fails. This is somewhat similar to my
>> real use-case:
>> <workflow-app name="test-sunil-wf" xmlns="uri:oozie:workflow:0.4">
>> <start to="test-echo-shell-1"/>
>> <action name="test-echo-shell-1">
>> <shell xmlns="uri:oozie:shell-action:0.1">
>> <job-tracker>${jobTracker}</job-tracker>
>> <name-node>${nameNode}</name-node>
>> <configuration>
>> <property>
>> <name>param1</name>
>> <value>${wf:id()}</value>
>> </property>
>> <property>
>> <name>param2</name>
>> <value>${wf:name()}</value>
>> </property>
>> <property>
>> <name>param3</name>
>> <value>${param1} and ${param2}</value>
>> </property>
>> </configuration>
>> <exec>/bin/echo</exec>
>> <argument>${param3}</argument>
>> <capture-output/>
>> </shell>
>> <ok to="end"/>
>> <error to="kill"/>
>> </action>
>> <kill name="kill">
>> <message>Action failed, error
>> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>> </kill>
>> <end name="end"/>
>> </workflow-app>
>>
>> The above XML fails with the same stacktrace as shown above.
>>
>> Thanks
>> Sunil
>
