[
https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425793#comment-13425793
]
Robert Joseph Evans commented on MAPREDUCE-4495:
------------------------------------------------
I really would encourage you to post a design document as well. I know it is
not very agile, but for big feature in Hadoop a lot of people prefer to see up
front what is going to happen, even if it is just a high level overview of how
you initially want to do this. If it change afterwards that is fine, but
people are going to want to understand what changed and why.
A feature like this is going to have a big impact on a lot of different people.
They want to be sure that they know how it is going to impact them, and they
want to have input on how the feature is done. Teams like Oozie, Pig, and Hive
are going to be directly impacted, and need to be treated as customers for a
feature like this. You can see from the comments that lots of people have
suggestions. I have a lot of suggestions too, but I would rather hold them back
until I see your design.
Thanks for doing this.
> Workflow Application Master in YARN
> -----------------------------------
>
> Key: MAPREDUCE-4495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha
> Reporter: Bo Wang
> Assignee: Bo Wang
>
> It is useful to have a workflow application master, which will be capable of
> running a DAG of jobs. The workflow client submits a DAG request to the AM
> and then the AM will manage the life cycle of this application in terms of
> requesting the needed resources from the RM, and starting, monitoring and
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master,
> these are some of the advantages:
> - Less number of consumed resources, since only one application master will
> be spawned for the whole workflow.
> - Reuse of resources, since the same resources can be used by multiple
> consecutive jobs in the workflow (no need to request/wait for resources for
> every individual job from the central RM).
> - More optimization opportunities in terms of collective resource requests.
> - Optimization opportunities in terms of rewriting and composing jobs in the
> workflow (e.g. pushing down Mappers).
> - This Application Master can be reused/extended by higher systems like Pig
> and hive to provide an optimized way of running their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
