[
https://issues.apache.org/jira/browse/OOZIE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13205154#comment-13205154
]
[email protected] commented on OOZIE-616:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3827/#review5000
-----------------------------------------------------------
Some more comments to make the error messages cleaner and usage of the
LauncherException tighter.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
<https://reviews.apache.org/r/3827/#comment11003>
Can we remove the default constructor and force developers to use one of
the other two constructors?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment10997>
Can you add a message here - fail("Expected to catch an exception but did
not encounter any") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment10998>
Can you add a message here - fail("Expected a LauncherException but
received an Exception") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment10999>
Can you add a message here - fail("Expected to catch an exception but did
not encounter any") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
<https://reviews.apache.org/r/3827/#comment11000>
Can you add a message here - fail("Expected a LauncherException but
received an Exception") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java
<https://reviews.apache.org/r/3827/#comment11002>
Can you add a message here - fail("Expected to catch an exception but did
not encounter any") ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java
<https://reviews.apache.org/r/3827/#comment11001>
Can you add a message here - fail("Expected a LauncherException but
received an Exception") ?
- Santhosh
On 2012-02-10 00:41:55, Virag Kothari wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3827/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-10 00:41:55)
bq.
bq.
bq. Review request for oozie, Santhosh Srinivasan and Alejandro Abdelnur.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. This is Kiran's patch. I am uploading the diff as he is not here. Original
diff at https://reviews.apache.org/r/3143/
bq. Thanks Alejandro for testing the patch out. The testcases are fixed now.
bq.
bq.
bq. This addresses bug oozie-616.
bq. https://issues.apache.org/jira/browse/oozie-616
bq.
bq.
bq. Diffs
bq. -----
bq.
bq.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
1242619
bq.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java
1242619
bq.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/FileSystemActions.java
PRE-CREATION
bq.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PrepareActionsDriver.java
PRE-CREATION
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestFileSystemActions.java
PRE-CREATION
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java
1242619
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestLauncher.java
1242619
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionError.java
1242619
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java
1242619
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java
1242619
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestPrepareActionsDriver.java
PRE-CREATION
bq.
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java
1242619
bq.
bq. Diff: https://reviews.apache.org/r/3827/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Ran the entire test suite
bq.
bq.
bq. Thanks,
bq.
bq. Virag
bq.
bq.
> Moving action prepare FS execution to LauncherMapper
> ----------------------------------------------------
>
> Key: OOZIE-616
> URL: https://issues.apache.org/jira/browse/OOZIE-616
> Project: Oozie
> Issue Type: Improvement
> Reporter: Kiran Nagasubramanian
>
> 1) Motivations:
> ----------------
> a) Currently Oozie Server executes the Prepare logic before posting the
> Launcher job to Hadoop. If the launcher fails and the execution is retried by
> Hadoop automatically, the prepare logic is not re-executed. So, for
> Java/MR/Pig actions, if the launcher fails and it is retried by Hadoop
> automatically, the prepare logic is not executed again. Once the prepare
> logic execution is moved to the Launcher, the prepare logic is also
> re-executed during every retry.
> b) Heavy duty operations like copy that are currently not supported can be
> supported once the prepare logic has been moved to the Launcher.
> 2) Design choices for the proposed model:
> ------------------------------------------------
> a) Separate classes for different types of actions
> ---------------------------------------------------------
> The different types of actions like FS actions, HCat related actions, etc.
> can be grouped together in separate classes like FSActions, HCatActions, etc.
> b) Passing the prepare logic to the Launcher
> ---------------------------------------------------
> Launcher needs the Prepare XML block to execute the actions. Oozie server can
> write the XML block to a file on DFS and then the Launcher could read from
> there.
> c) Execution of actions through a "Driver"
> -----------------------------------------------
> The Launcher can pass the XML block to the Driver which parses the content
> and calls corresponding methods that are grouped in different classes.
> 3) Pros and Cons of the design change:
> --------------------------------------------
> a) User facing impacts:
> -------------------------
> i) At present: In case of prepare block failure, there is no impact on the
> retry of map reduce job since the prepare block would have got executed
> earlier itself. The impact is seen only when Launcher fails.
>
> As per the proposed design: When the prepare block fails it will impact the
> retry of the map reduce job.
>
> ii) At present: In the case of prepare block failure, the
> launcher mapper is not launched at all.
>
> As per the proposed design: The cost of executing the
> prepare block would go up since it requires the launch of a map only task.
> This is highly pronounced in case of failures
>
> Thanks a lot for all the input from Oozie team members.
> Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
