[
https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bo Wang updated MAPREDUCE-4495:
-------------------------------
Attachment: MAPREDUCE-4495-v1.1.patch
Did some small changes to include MR jars.
Following are logs from Workflow AM engine:
2/09/17 11:23:56 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [#start#]
12/09/17 11:23:56 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Exit
node [#start#] transitions [JobA]
12/09/17 11:23:56 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [JobA]
12/09/17 11:24:05 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Exit
node [JobA] transitions [forking]
12/09/17 11:24:05 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [forking]
12/09/17 11:24:05 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Exit
node [forking] transitions [JobB,JobC]
12/09/17 11:24:05 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [JobB]
12/09/17 11:24:05 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [JobC]
12/09/17 11:24:13 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Exit
node [JobB] transitions [joining]
12/09/17 11:24:13 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [joining]
12/09/17 11:24:15 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Exit
node [JobC] transitions [joining]
12/09/17 11:24:15 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [joining]
12/09/17 11:24:15 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Exit
node [joining] transitions [JobD]
12/09/17 11:24:15 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [JobD]
12/09/17 11:24:23 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Exit
node [JobD] transitions [end]
12/09/17 11:24:23 [AsyncDispatcher event handler] DEBUG wflib.Workflow: Enter
node [end]
> Workflow Application Master in YARN
> -----------------------------------
>
> Key: MAPREDUCE-4495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha
> Reporter: Bo Wang
> Assignee: Bo Wang
> Attachments: MAPREDUCE-4495-v1.1.patch, MAPREDUCE-4495-v1.patch,
> MapReduceWorkflowAM.pdf
>
>
> It is useful to have a workflow application master, which will be capable of
> running a DAG of jobs. The workflow client submits a DAG request to the AM
> and then the AM will manage the life cycle of this application in terms of
> requesting the needed resources from the RM, and starting, monitoring and
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master,
> these are some of the advantages:
> - Less number of consumed resources, since only one application master will
> be spawned for the whole workflow.
> - Reuse of resources, since the same resources can be used by multiple
> consecutive jobs in the workflow (no need to request/wait for resources for
> every individual job from the central RM).
> - More optimization opportunities in terms of collective resource requests.
> - Optimization opportunities in terms of rewriting and composing jobs in the
> workflow (e.g. pushing down Mappers).
> - This Application Master can be reused/extended by higher systems like Pig
> and hive to provide an optimized way of running their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
