Hi all,
I'm not sure how many people on this list also follow the #oozie IRC
channel, so here goes.
I found out yesterday that, unfortunately, the coord:conf EL function
was not implemented, meaning that it was impossible to refer to
properties that were not valid Java identifiers. For example, I want to
define the start and end times of my coordinators using two properties
that are passed via the command-line when the coordinator job is
submitted to Oozie:
$ cat coordinator.xml
<coordinator-app name="test"
start="${coord:conf(bob.schedule.start)}"
end="${coord:conf(bob.schedule.end)}"
timezone="US/Pacific">
<action>
<workflow>
<app-path>${coord:conf(bob.app.path)}</app-path>
</workflow>
</action>
</coordinator>
$ ~/opt/oozie/bin/oozie -config .../job.properties \
-Dbob.schedule.start=2012-03-20T11:15Z \
-Dbob.schedule.end=2012-03-27T00:00Z \
-run
With job.properties containing a few other configuration properties,
including of course bob.app.path (the path of the workflow application
in HDFS).
A similar function, wf:conf(), already exists for workflows, and is
implemented in DagELFunctions.java#wf_conf(String property): it gets the
Configuration object from the workflow instance, and accesses said
property by name.
I'm trying to implement the same functionality for coordinators, but I
can't figure out how to access the configuration from
CoordELFunctions.java. It's not easy to deep-dive into such a codebase
like this. From what I can gather, the job configuration properties are
set as variables into the ELEvaluator in
CoordELEvaluator.java#setConfigToEval(), which is called when the
ELEvaluator is created.
So I thought all I had to do was to implement coord_conf in
CoordELFunctions.java that gets variables from the ELEvaluator:
/**
* Return a job configuration property for the coordinator.
*
* @param property property name.
* @return the value of the property, <code>null</code> if the property is
undefined.
*/
public static String coord_conf(String property) {
ELEvaluator eval = ELEvaluator.getCurrent();
return (String) eval.getVariable(property);
}
Then declare it in oozie-default.xml. Unfortunately, even though it seems like
my coord_conf function is called correctly, it doesn't seem like the variable
is resolved correctly. This in what I get in the Oozie log when I try all this:
Caused by: java.lang.Exception: Unable to evaluate
:${coord:conf(bob.schedule.start)}:
at
org.apache.oozie.coord.CoordELFunctions.evalAndWrap(CoordELFunctions.java:551)
at
org.apache.oozie.command.coord.CoordSubmitXCommand.resolveAttribute(CoordSubmitXCommand.java:848)
... 33 more
Caused by: javax.servlet.jsp.el.ELException: variable [bob] cannot be resolved
at
org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:106)
I also don't understand why the variable name here is 'bob' and not
'bob.schedule.start'. And if I add -Dbob=foo to the command-line, the
exception is a bit different:
Caused by: javax.servlet.jsp.el.ELException: Unable to find a value for
"schedule" in object of class "java.lang.String" using operator "."
Which makes me think that maybe I don't understand at all how variables
in ELEvaluator work ;-)
If anyone more familiar with the Oozie codebase or how this particular
area works could shed some light on what's going on here, I'd be very
grateful! And hopefully I can wrap this thing up and send a proper
patch to contribute to Oozie.
In the meantime, my current, incomplete patch is attached.
Thanks in advance,
/Maxime
--
Maxime Petazzoni, Platform Engineer at Turn, Inc (www.turn.com)
Index: core/src/test/resources/coord-action-get.xml
===================================================================
--- core/src/test/resources/coord-action-get.xml (revision 1302703)
+++ core/src/test/resources/coord-action-get.xml (working copy)
@@ -39,7 +39,7 @@
</output-events>
<action>
<workflow>
- <app-path>${wfAppPath}</app-path>
+ <app-path>${coord:conf(wfAppPath)}</app-path>
<configuration>
<property>
<name>jobTracker</name>
Index: core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java
===================================================================
--- core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java (revision 1302703)
+++ core/src/test/java/org/apache/oozie/coord/TestCoordELFunctions.java (working copy)
@@ -815,6 +815,13 @@
assertEquals("mycoordinator-app", CoordELFunctions.evalAndWrap(eval, expr));
}
+ public void testConf() throws Exception {
+ init("coord-job-submit-freq");
+ eval.setVariable("my.test.variable", "test");
+ String expr = "${coord:conf(my.test.variable)";
+ assertEquals("test", CoordELFunctions.evalAndWrap(eval, expr));
+ }
+
public void testUser() throws Exception {
init("coord-job-submit-freq");
String expr = "${coord:user()}";
Index: core/src/main/resources/oozie-default.xml
===================================================================
--- core/src/main/resources/oozie-default.xml (revision 1302703)
+++ core/src/main/resources/oozie-default.xml (working copy)
@@ -548,6 +548,7 @@
coord:minutes=org.apache.oozie.coord.CoordELFunctions#ph1_coord_minutes,
coord:endOfDays=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfDays,
coord:endOfMonths=org.apache.oozie.coord.CoordELFunctions#ph1_coord_endOfMonths,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
@@ -593,7 +594,10 @@
<property>
<name>oozie.service.ELService.functions.coord-job-submit-nofuncs</name>
- <value>coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user</value>
+ <value>
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
+ coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
+ </value>
<description>
EL functions declarations, separated by commas, format is [PREFIX:]NAME=CLASS#METHOD.
</description>
@@ -639,6 +643,7 @@
coord:latest=org.apache.oozie.coord.CoordELFunctions#ph1_coord_latest_echo,
coord:future=org.apache.oozie.coord.CoordELFunctions#ph1_coord_future_echo,
coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
@@ -689,6 +694,7 @@
coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_formatTime_echo,
coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph1_coord_actionId_echo,
coord:name=org.apache.oozie.coord.CoordELFunctions#ph1_coord_name_echo,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
@@ -734,6 +740,7 @@
<name>oozie.service.ELService.functions.coord-sla-submit</name>
<value>
coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph1_coord_nominalTime_echo_fixed,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
@@ -783,6 +790,7 @@
coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph2_coord_actionId,
coord:name=org.apache.oozie.coord.CoordELFunctions#ph2_coord_name,
coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
@@ -832,6 +840,7 @@
coord:latest=org.apache.oozie.coord.CoordELFunctions#ph2_coord_latest_echo,
coord:future=org.apache.oozie.coord.CoordELFunctions#ph2_coord_future_echo,
coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_formatTime,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
@@ -876,6 +885,7 @@
<name>oozie.service.ELService.functions.coord-sla-create</name>
<value>
coord:nominalTime=org.apache.oozie.coord.CoordELFunctions#ph2_coord_nominalTime,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
@@ -929,6 +939,7 @@
coord:formatTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_formatTime,
coord:actionId=org.apache.oozie.coord.CoordELFunctions#ph3_coord_actionId,
coord:name=org.apache.oozie.coord.CoordELFunctions#ph3_coord_name,
+ coord:conf=org.apache.oozie.coord.CoordELFunctions#coord_conf,
coord:user=org.apache.oozie.coord.CoordELFunctions#coord_user
</value>
<description>
Index: core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java
===================================================================
--- core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java (revision 1302703)
+++ core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java (working copy)
@@ -1049,6 +1049,17 @@
}
/**
+ * Return a job configuration property for the coordinator.
+ *
+ * @param property property name.
+ * @return the value of the property, <code>null</code> if the property is undefined.
+ */
+ public static String coord_conf(String property) {
+ ELEvaluator eval = ELEvaluator.getCurrent();
+ return (String) eval.getVariable(property);
+ }
+
+ /**
* Return the user that submitted the coordinator job.
*
* @return the user that submitted the coordinator job.
