Moving action prepare FS execution to LauncherMapper
----------------------------------------------------
Key: OOZIE-616
URL: https://issues.apache.org/jira/browse/OOZIE-616
Project: Oozie
Issue Type: Improvement
Reporter: Kiran Nagasubramanian
1) Motivations:
----------------
a) Currently Oozie Server executes the Prepare logic before posting the
Launcher job to Hadoop. If the launcher fails and the execution is retried by
Hadoop automatically, the prepare logic is not re-executed. So, for Java/MR/Pig
actions, if the launcher fails and it is retried by Hadoop automatically, the
prepare logic is not executed again. Once the prepare logic execution is moved
to the Launcher, the prepare logic is also re-executed during every retry.
b) Heavy duty operations like copy that are currently not supported can be
supported once the prepare logic has been moved to the Launcher.
2) Design choices for the proposed model:
------------------------------------------------
a) Separate classes for different types of actions
---------------------------------------------------------
The different types of actions like FS actions, HCat related actions, etc.
can be grouped together in separate classes like FSActions, HCatActions, etc.
b) Passing the prepare logic to the Launcher
---------------------------------------------------
Launcher needs the Prepare XML block to execute the actions. Oozie server can
write the XML block to a file on DFS and then the Launcher could read from
there.
c) Execution of actions through a "Driver"
-----------------------------------------------
The Launcher can pass the XML block to the Driver which parses the content
and calls corresponding methods that are grouped in different classes.
3) Pros and Cons of the design change:
--------------------------------------------
a) User facing impacts:
-------------------------
i) At present: In case of prepare block failure, there is no impact on the
retry of map reduce job since the prepare block would have got executed earlier
itself. The impact is seen only when Launcher fails.
As per the proposed design: When the prepare block fails it will impact the
retry of the map reduce job.
ii) At present: In the case of prepare block failure, the
launcher mapper is not launched at all.
As per the proposed design: The cost of executing the
prepare block would go up since it requires the launch of a map only task. This
is highly pronounced in case of failures
Thanks a lot for all the input from Oozie team members.
Please post your suggestions. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
