Yesha Vora updated YARN-8283:
    Attachment: [Design Doc] [YARN-8283] MaWo - A Master Worker framework on 
top of YARN Services.pdf

> [Umbrella] MaWo - A Master Worker framework on top of YARN Services
> -------------------------------------------------------------------
>                 Key: YARN-8283
>                 URL: https://issues.apache.org/jira/browse/YARN-8283
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Yesha Vora
>            Assignee: Yesha Vora
>            Priority: Major
>         Attachments: [Design Doc] [YARN-8283] MaWo - A Master Worker 
> framework on top of YARN Services.pdf
> There is a need for an application / framework to handle Master-Worker 
> scenarios. There are existing frameworks on YARN which can be used to run a 
> job in distributed manner such as Mapreduce, Tez, Spark etc. But 
> master-worker use-cases usually are force-fed into one of these existing 
> frameworks which have been designed primarily around data-parallelism instead 
> of generic Master Worker type of computations.
> In this JIRA, we’d like to contribute MaWo - a YARN Service based framework 
> that achieves this goal. The overall goal is to create an app that can take 
> an input job specification with tasks, their durations and have a Master dish 
> the tasks off to a predetermined set of workers. The components will be 
> responsible for making sure that the tasks and the overall job finish in 
> specific time durations.
> We have been using a version of the MaWo framework for running unit tests of 
> Hadoop in a parallel manner on an existing Hadoop YARN cluster. What 
> typically takes 10 hours to run all of Hadoop project’s unit-tests can finish 
> under 20 minutes on a MaWo app of about 50 containers!
> YARN-3307 was an original attempt at this but through a first-class YARN app. 
> In this JIRA, we instead use YARN Service for orchestration so that our code 
> can focus on the core Master Worker paradigm.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to