[
https://issues.apache.org/jira/browse/OOZIE-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193424#comment-13193424
]
[email protected] commented on OOZIE-636:
-----------------------------------------------------
bq. On 2012-01-23 22:44:03, Alejandro Abdelnur wrote:
bq. >
trunk/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowApp.java,
line 117
bq. > <https://reviews.apache.org/r/3486/diff/1/?file=68840#file68840line117>
bq. >
bq. > this logic should be in the parser it self with the rest of the
validation logic
moved
bq. On 2012-01-23 22:44:03, Alejandro Abdelnur wrote:
bq. >
trunk/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowApp.java,
line 118
bq. > <https://reviews.apache.org/r/3486/diff/1/?file=68840#file68840line118>
bq. >
bq. > The algorithm seems a bit too complex.
bq. >
bq. > How about the following approach:
bq. >
bq. > * as part of the current validation traversal all fork&join nodes
would be harvested in 2 sets (forksSet and joinsSet).
bq. >
bq. > * after doing the current validation the 2 sets above will be
populated.
bq. >
bq. > * first check is that the size of both sets is the same.
bq. >
bq. > * if the forksSet is > 0 then
bq. >
bq. > ** get a fork node
bq. > ** follow the first path until a join or fork is reached
bq. > *** if a join is reached remember the join node and remove it from
the joinsSet.
bq. > *** if a fork is reached start this algorithm recursively
bq. > ** follow every subsequent path
bq. > *** if a fork is reached start this algorithm recursively
bq. > *** if a join is reached ensure it is the same join of the first
path.
bq. >
Yes..It seems better in terms of code complexity. Good suggestion, thanks!
- Virag
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3486/#review4543
-----------------------------------------------------------
On 2012-01-25 23:00:15, Virag Kothari wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3486/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-01-25 23:00:15)
bq.
bq.
bq. Review request for oozie, Mohammad Islam and Angelo K. Huang.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Validate fork and join at wf submission time
bq. https://issues.apache.org/jira/browse/OOZIE-636
bq.
bq. Brief description of algo:
bq.
bq. A modified dfs algorithm is used. Two stacks, one for dfs traversal and
other for maintaining fork join status, are kept. When a fork is encountered
during traversal, it is added to the forkjoin stack and number of paths
associated with the fork is also stored. When a node’s child is seen as a
join, the join is added to the forkJoin stack and the no. of paths to it is
updated. When the number of paths for fork and join are equal, then the
fork/join pair is removed from the forkJoin stack and join is pushed to the
dfsStack.
bq.
bq. Nodes other than fork and join are only pushed to the dfs stack.
bq. If a action node is seen, only the node's "ok-to" transition is considered
bq.
bq.
bq. While(!stack.isEmpty()){
bq. Node n = DfsStack.pop()
bq. n.traversed = true;
bq. If(n.type==fork){
bq. ForkJoinStack.push(new Element(n, n.paths) );
bq. }
bq. List<Node> childs = getUnvisitedChildnodes(n)
bq. For(Node n: childs){
bq. If (n.type==join){
bq. Boolean b=isForkJoinCleared(ForkJoinStack)
bq. If(!b){
bq. Continue;
bq. }
bq. stack.push(n);
bq. n.traversed = true;
bq. }
bq. }
bq.
bq.
bq. This addresses bug OOZIE-636.
bq. https://issues.apache.org/jira/browse/OOZIE-636
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/core/src/main/java/org/apache/oozie/ErrorCode.java 1235973
bq.
trunk/core/src/main/java/org/apache/oozie/workflow/lite/LiteWorkflowAppParser.java
1235973
bq.
trunk/core/src/test/java/org/apache/oozie/service/TestLiteWorkflowAppService.java
1235973
bq.
trunk/core/src/test/java/org/apache/oozie/workflow/lite/TestLiteWorkflowAppParser.java
1235973
bq. trunk/core/src/test/resources/wf-schema-valid.xml 1235973
bq.
bq. Diff: https://reviews.apache.org/r/3486/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Test case to validate fork-join added
bq.
bq.
bq. Thanks,
bq.
bq. Virag
bq.
bq.
> Check fork and join in the workflow in the submission time
> -----------------------------------------------------------
>
> Key: OOZIE-636
> URL: https://issues.apache.org/jira/browse/OOZIE-636
> Project: Oozie
> Issue Type: Bug
> Reporter: Virag Kothari
>
> Enhancement: Oozie should check that the fork node and join node are correct
> in pair when user submits the job. This should be a static check, not when
> the workflow is running.
> Current logic bug:
> A workflow with different number of forks and joins was run. The wf job
> should have been killed but it succeeded. Also, strangely, the action was
> killed.
> Following are the different types of tests run and their results with varying
> delays.
> test1: wf job SUCCEEDED, action java12 KILLED.
> delay11=11
> delay12=12
> delay121=1
> delay122=2
> delay21=1
> delay22=1
> test2: wf job SUCCEEDED, action java12 KILLED.
> delay11=1
> delay12=12
> delay121=1
> delay122=2
> delay21=1
> delay22=1
> test3: wf job SUCCEEED, all actions OK. question: why wf job always pass in
> this scenario, even when fork-join not in
> pair?
> delay11=10
> delay12=10
> delay121=15
> delay122=15
> delay21=20
> delay22=20
> workflow.xml
> ============
> <workflow-app xmlns='uri:oozie:workflow:0.1' name='fork-join-4735180-wf'>
> <start to='fork1' />
> <fork name="fork1">
> <path start="java11" />
> <path start="fork12" />
> </fork>
> <action name='java11'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay11}</arg>
> </java>
> <ok to="java12" />
> <error to="fail" />
> </action>
> <action name='java12'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay12}</arg>
> </java>
> <ok to="join1" />
> <error to="fail" />
> </action>
> <fork name="fork12">
> <path start="java121" />
> <path start="java122" />
> </fork>
> <action name='java121'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay121}</arg>
> </java>
> <ok to="join12" />
> <error to="fail" />
> </action>
> <action name='java122'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay122}</arg>
> </java>
> <ok to="join12" />
> <error to="fail" />
> </action>
> <join name="join12" to="fork2" />
> <fork name="fork2">
> <path start="java21" />
> <path start="java22" />
> </fork>
> <action name='java21'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay21}</arg>
> </java>
> <ok to="join1" />
> <error to="fail" />
> </action>
> <action name='java22'>
> <java>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> </configuration>
> <main-class>qa.test.tests.testsleep</main-class>
> <arg>${delay22}</arg>
> </java>
> <ok to="join1" />
> <error to="fail" />
> </action>
> <join name="join1" to="end" />
> <kill name="fail">
> <message>Streaming Map/Reduce failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name='end' />
> </workflow-app>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
