[
https://issues.apache.org/jira/browse/OOZIE-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191576#comment-13191576
]
[email protected] commented on OOZIE-636:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3486/#review4543
-----------------------------------------------------------
trunk/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowApp.java
<https://reviews.apache.org/r/3486/#comment10135>
this logic should be in the parser it self with the rest of the validation
logic
trunk/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowApp.java
<https://reviews.apache.org/r/3486/#comment10136>
The algorithm seems a bit too complex.
How about the following approach:
* as part of the current validation traversal all fork&join nodes would be
harvested in 2 sets (forksSet and joinsSet).
* after doing the current validation the 2 sets above will be populated.
* first check is that the size of both sets is the same.
* if the forksSet is > 0 then
** get a fork node
** follow the first path until a join or fork is reached
*** if a join is reached remember the join node and remove it from the
joinsSet.
*** if a fork is reached start this algorithm recursively
** follow every subsequent path
*** if a fork is reached start this algorithm recursively
*** if a join is reached ensure it is the same join of the first path.
- Alejandro
On 2012-01-13 17:39:11, Virag Kothari wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3486/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-01-13 17:39:11)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Validate fork and join at wf submission time
bq. https://issues.apache.org/jira/browse/OOZIE-636
bq.
bq. Brief description of algo:
bq.
bq. A modified dfs algorithm is used. Two stacks, one for dfs traversal and
other for maintaining fork join status, are kept. When a fork is encountered
during traversal, it is added to the forkjoin stack and number of paths
associated with the fork is also stored. When a node’s child is seen as a
join, the join is added to the forkJoin stack and the no. of paths to it is
updated. When the number of paths for fork and join are equal, then the
fork/join pair is removed from the forkJoin stack and join is pushed to the
dfsStack.
bq.
bq. Nodes other than fork and join are only pushed to the dfs stack.
bq. If a action node is seen, only the node's "ok-to" transition is considered
bq.
bq.
bq. While(!stack.isEmpty()){
bq. Node n = DfsStack.pop()
bq. n.traversed = true;
bq. If(n.type==fork){
bq. ForkJoinStack.push(new Element(n, n.paths) );
bq. }
bq. List<Node> childs = getUnvisitedChildnodes(n)
bq. For(Node n: childs){
bq. If (n.type==join){
bq. Boolean b=isForkJoinCleared(ForkJoinStack)
bq. If(!b){
bq. Continue;
bq. }
bq. stack.push(n);
bq. n.traversed = true;
bq. }
bq. }
bq.
bq.
bq. This addresses bug OOZIE-636.
bq. https://issues.apache.org/jira/browse/OOZIE-636
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/ErrorCode.java 1230856
bq.
trunk/core/src/main/java/org/apache/oozie/workflow/lite/ForkJoinElement.java
PRE-CREATION
bq.
trunk/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowApp.java
1230856
bq.
trunk/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java
1230856
bq.
trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java
1230856
bq.
trunk/core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowApp.java
PRE-CREATION
bq. trunk/core/src/test/resources/wf-schema-valid.xml 1230856
bq.
bq. Diff: https://reviews.apache.org/r/3486/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test case to validate fork-join added
bq.
bq.
bq. Thanks,
bq.
bq. Virag
bq.
bq.
> Check fork and join in the workflow in the submission time
> -----------------------------------------------------------
>
> Key: OOZIE-636
> URL: https://issues.apache.org/jira/browse/OOZIE-636
> Project: Oozie
> Issue Type: Bug
> Reporter: Virag Kothari
>
> Enhancement: Oozie should check that the fork node and join node are correct
> in pair when user submits the job. This should be a static check, not when
> the workflow is running.
> Current logic bug:
> A workflow with different number of forks and joins was run. The wf job
> should have been killed but it succeeded. Also, strangely, the action was
> killed.
> Following are the different types of tests run and their results with varying
> delays.
> test1: wf job SUCCEEDED, action java12 KILLED.
> delay11=11
> delay12=12
> delay121=1
> delay122=2
> delay21=1
> delay22=1
> test2: wf job SUCCEEDED, action java12 KILLED.
> delay11=1
> delay12=12
> delay121=1
> delay122=2
> delay21=1
> delay22=1
> test3: wf job SUCCEEED, all actions OK. question: why wf job always pass in
> this scenario, even when fork-join not in
> pair?
> delay11=10
> delay12=10
> delay121=15
> delay122=15
> delay21=20
> delay22=20
> workflow.xml
> ============
> <workflow-app xmlns='uri:oozie:workflow:0.1' name='fork-join-4735180-wf'>
> <start to='fork1' />
> <fork name="fork1">
> <path start="java11" />
> <path start="fork12" />
> </fork>
> <action name='java11'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay11}</arg>
> </java>
> <ok to="java12" />
> <error to="fail" />
> </action>
> <action name='java12'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay12}</arg>
> </java>
> <ok to="join1" />
> <error to="fail" />
> </action>
> <fork name="fork12">
> <path start="java121" />
> <path start="java122" />
> </fork>
> <action name='java121'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay121}</arg>
> </java>
> <ok to="join12" />
> <error to="fail" />
> </action>
> <action name='java122'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay122}</arg>
> </java>
> <ok to="join12" />
> <error to="fail" />
> </action>
> <join name="join12" to="fork2" />
> <fork name="fork2">
> <path start="java21" />
> <path start="java22" />
> </fork>
> <action name='java21'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay21}</arg>
> </java>
> <ok to="join1" />
> <error to="fail" />
> </action>
> <action name='java22'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay22}</arg>
> </java>
> <ok to="join1" />
> <error to="fail" />
> </action>
> <join name="join1" to="end" />
> <kill name="fail">
> <message>Streaming Map/Reduce failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name='end' />
> </workflow-app>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
