Hi Robert,
Thanks for the pointing to OOZIE-1978, this will be helpful.
PFA my workflows. I have 2 workflows and both of them fails validation
with different error message :-
* not-workflow.xml fails due to "E0730: Fork/Joins not in pair². Node ³d²
never executes if I submit with disabled validation.
* working.xml which fails due to "E0742: No Fork for join (Join_f) to pair
with". This workflow works perfectly if I submit with disabled validation.
Can you please share more details regarding why do we need Fork/Join to be
in pair, not-workflow.xml have lesser Fork because of a 3-way fork at
starting. And why Oozie never executes node ³d² ?
Similarly why should we have paired Fork/Join in each path from start to
end node and how working.xml executes perfectly ? User¹s workflow may be
such that logically graph has unpaired fork/join ?
Thanks,
Gaurav
________________________________________
From: Robert Kanter [[email protected]]
Sent: Wednesday, February 18, 2015 6:33 PM
To: [email protected]
Subject: Re: Issues with disabling ForkJoin Validation
Hi Gaurav,
IIRC, it should say why it's failing the ForkJoin validation. What does it
say?
There are two reasons why we allow disabling ForkJoin validation:
1. In case there's a bug in the ForkJoin validation code (it's very
complicated), there's a workaround to allow skipping it
2. In larger workflows, the ForkJoin validation code can take a very long
time to finish (on the order of hours or days), so there needs to be a way
to skip it. (OOZIE-1978 <https://issues.apache.org/jira/browse/OOZIE-1978>
is open to improve this)
As for whether or not it's safe to disable the validation, if the workflow
is working fine, then it's okay. The main purpose of the ForkJoin
validation is to fail incorrect workflows at submission time, rather than
during runtime. I believe that there are some technically incorrect
ForkJoin cases where Oozie can still work just fine, even though it's
wrong. You've either done that or found a bug in the ForkJoin validation
code :)
Can you post the reason and the workflow so I can see what it's complaining
about?
- Robert
On Wed, Feb 18, 2015 at 5:15 PM, Gaurav Aggarwal <
[email protected]> wrote:
> Hi,
>
> I have a workflow which fails ForkJoin validation in oozie but works fine
> if I set oozie.wf.validate.ForkJoin=false.
>
> Is it safe to set oozie.wf.validate.ForkJoin=false ?
>
> I want to understand potential runtime issues that can be created if I
> disable fork join validation.
>
> Thanks,
> Gaurav
>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<workflow-app xmlns="uri:oozie:workflow:0.4" name="az:f">
<start to="k"/>
<action name="k">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : k</description>
</property>
</configuration>
<exec>k.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin2/13/f/2/temp/k.sh</file>
</shell:shell>
<ok to="FORK_k"/>
<error to="ERROR"/>
</action>
<fork name="FORK_k">
<path start="d"/>
<path start="a"/>
</fork>
<action name="a">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : a</description>
</property>
</configuration>
<exec>a.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin2/13/f/2/temp/a.sh</file>
</shell:shell>
<ok to="FORK_a"/>
<error to="ERROR"/>
</action>
<fork name="FORK_a">
<path start="b"/>
<path start="c"/>
</fork>
<action name="c">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : c</description>
</property>
</configuration>
<exec>c.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin2/13/f/2/temp/c.sh</file>
</shell:shell>
<ok to="JOIN_e"/>
<error to="ERROR"/>
</action>
<action name="d">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : d</description>
</property>
</configuration>
<exec>d.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin2/13/f/2/temp/d.sh</file>
</shell:shell>
<ok to="JOIN_e"/>
<error to="ERROR"/>
</action>
<action name="b">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : b</description>
</property>
</configuration>
<exec>b.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin2/13/f/2/temp/b.sh</file>
</shell:shell>
<ok to="JOIN_f"/>
<error to="ERROR"/>
</action>
<action name="e">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : e</description>
</property>
</configuration>
<exec>e.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin2/13/f/2/temp/e.sh</file>
</shell:shell>
<ok to="JOIN_f"/>
<error to="ERROR"/>
</action>
<join to="e" name="JOIN_e"/>
<action name="f">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : f</description>
</property>
</configuration>
<exec>f.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin2/13/f/2/temp/f.sh</file>
</shell:shell>
<ok to="END"/>
<error to="ERROR"/>
</action>
<join to="f" name="JOIN_f"/>
<kill name="ERROR">
<message>Error in workflow!</message>
</kill>
<end name="END"/>
</workflow-app>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<workflow-app xmlns="uri:oozie:workflow:0.4" name="az:e">
<start to="FORK_START"/>
<fork name="FORK_START">
<path start="b"/>
<path start="c"/>
<path start="a"/>
</fork>
<action name="c">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : c</description>
</property>
</configuration>
<exec>c.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin/11/e/1/temp/c.sh</file>
</shell:shell>
<ok to="JOIN_d"/>
<error to="ERROR"/>
</action>
<action name="b">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : b</description>
</property>
</configuration>
<exec>b.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin/11/e/1/temp/b.sh</file>
</shell:shell>
<ok to="JOIN_d"/>
<error to="ERROR"/>
</action>
<action name="a">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : a</description>
</property>
</configuration>
<exec>a.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin/11/e/1/temp/a.sh</file>
</shell:shell>
<ok to="JOIN_e"/>
<error to="ERROR"/>
</action>
<action name="d">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : d</description>
</property>
</configuration>
<exec>d.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin/11/e/1/temp/d.sh</file>
</shell:shell>
<ok to="JOIN_e"/>
<error to="ERROR"/>
</action>
<join to="d" name="JOIN_d"/>
<action name="e">
<shell:shell xmlns:shell="uri:oozie:shell-action:0.3" xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
<description>Shell Action for Job : e</description>
</property>
</configuration>
<exec>e.sh</exec>
<file>/user/gaggarwa/oozie/forkJoin/11/e/1/temp/e.sh</file>
</shell:shell>
<ok to="END"/>
<error to="ERROR"/>
</action>
<join to="e" name="JOIN_e"/>
<kill name="ERROR">
<message>Error in workflow!</message>
</kill>
<end name="END"/>
</workflow-app>
