Yesha Vora created YARN-8283:
--------------------------------
Summary: [Umbrella] MaWo - A Master Worker framework on top of
YARN Services
Key: YARN-8283
URL: https://issues.apache.org/jira/browse/YARN-8283
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Yesha Vora
There is a need for an application / framework to handle Master-Worker
scenarios. There are existing frameworks on YARN which can be used to run a job
in distributed manner such as Mapreduce, Tez, Spark etc. But master-worker
use-cases usually are force-fed into one of these existing frameworks which
have been designed primarily around data-parallelism instead of generic Master
Worker type of computations.
In this JIRA, we’d like to contribute MaWo - a YARN Service based framework
that achieves this goal. The overall goal is to create an app that can take an
input job specification with tasks, their durations and have a Master dish the
tasks off to a predetermined set of workers. The components will be responsible
for making sure that the tasks and the overall job finish in specific time
durations.
We have been using a version of the MaWo framework for running unit tests of
Hadoop in a parallel manner on an existing Hadoop YARN cluster. What typically
takes 10 hours to run all of Hadoop project’s unit-tests can finish under 20
minutes on a MaWo app of about 50 containers!
YARN-3307 was an original attempt at this but through a first-class YARN app.
In this JIRA, we instead use YARN Service for orchestration so that our code
can focus on the core Master Worker paradigm.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
