[
https://issues.apache.org/jira/browse/YARN-8283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511791#comment-16511791
]
Yesha Vora commented on YARN-8283:
----------------------------------
Design doc is attached.
> [Umbrella] MaWo - A Master Worker framework on top of YARN Services
> -------------------------------------------------------------------
>
> Key: YARN-8283
> URL: https://issues.apache.org/jira/browse/YARN-8283
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Yesha Vora
> Assignee: Yesha Vora
> Priority: Major
> Attachments: [Design Doc] [YARN-8283] MaWo - A Master Worker
> framework on top of YARN Services.pdf
>
>
> There is a need for an application / framework to handle Master-Worker
> scenarios. There are existing frameworks on YARN which can be used to run a
> job in distributed manner such as Mapreduce, Tez, Spark etc. But
> master-worker use-cases usually are force-fed into one of these existing
> frameworks which have been designed primarily around data-parallelism instead
> of generic Master Worker type of computations.
> In this JIRA, we’d like to contribute MaWo - a YARN Service based framework
> that achieves this goal. The overall goal is to create an app that can take
> an input job specification with tasks, their durations and have a Master dish
> the tasks off to a predetermined set of workers. The components will be
> responsible for making sure that the tasks and the overall job finish in
> specific time durations.
> We have been using a version of the MaWo framework for running unit tests of
> Hadoop in a parallel manner on an existing Hadoop YARN cluster. What
> typically takes 10 hours to run all of Hadoop project’s unit-tests can finish
> under 20 minutes on a MaWo app of about 50 containers!
> YARN-3307 was an original attempt at this but through a first-class YARN app.
> In this JIRA, we instead use YARN Service for orchestration so that our code
> can focus on the core Master Worker paradigm.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
