[
https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427297#comment-13427297
]
Chris A. Mattmann commented on MAPREDUCE-4495:
----------------------------------------------
Hi Alejandro:
Thanks for your reply. Saw your comment about:
bq. Thus we would not get distracted with the bureaucracy required to bootstrap
an incubator project.
It's hard to respond to this one without specifics. At best I can say the
bureaucracy in an incubator project isn't really more than the bureaucracy in
the Hadoop project, or compared to Apache as a whole.
As for:
bq. Doing it in Oozie means that Pig/Hive would not be able to consume it
easily as it would create a circular dependency among those projects.
Circular dependencies are technical challenges that can be overcome via a
number of ways (including maintaining duplicate code from the upstream project
in the downstream project and removing the direct dependency until the
specifics of the separation can be fully worked out).
My question to you: do you think that this code:
# has its own community?
# has its own release cycle?
# may release its own software products/artifacts?
If so, then my recommendation is to consider starting a new Incubator project.
If not, well that's something to
continue discussing here.
> Workflow Application Master in YARN
> -----------------------------------
>
> Key: MAPREDUCE-4495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha
> Reporter: Bo Wang
> Assignee: Bo Wang
>
> It is useful to have a workflow application master, which will be capable of
> running a DAG of jobs. The workflow client submits a DAG request to the AM
> and then the AM will manage the life cycle of this application in terms of
> requesting the needed resources from the RM, and starting, monitoring and
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master,
> these are some of the advantages:
> - Less number of consumed resources, since only one application master will
> be spawned for the whole workflow.
> - Reuse of resources, since the same resources can be used by multiple
> consecutive jobs in the workflow (no need to request/wait for resources for
> every individual job from the central RM).
> - More optimization opportunities in terms of collective resource requests.
> - Optimization opportunities in terms of rewriting and composing jobs in the
> workflow (e.g. pushing down Mappers).
> - This Application Master can be reused/extended by higher systems like Pig
> and hive to provide an optimized way of running their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
