[
https://issues.apache.org/jira/browse/HADOOP-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154517#comment-16154517
]
Subru Krishnan commented on HADOOP-14840:
-----------------------------------------
[~vrushalic], thanks for your comments, please find my response inline.
bq. This sounds interesting. Will there be subtasks under this?
We have developed this tool and are using it internally. So it has already been
through rigorous reviews etc so was just planning on a single uber patch to
save time/efficiency. We can break it up if really required.
bq. Similar estimations for MR jobs like reducer estimation & memory estimation
are done based on past runs using a tool called hRaven
(https://github.com/twitter/hraven)
We are aware of hRaven. In fact, we have a similar work for Hive and Tez called
[PerfOrator|http://dl.acm.org/citation.cfm?id=2987566]. With the ever growing
frameworks (and optimizations/updates to existing ones), we decided to follow a
different approach here:
* Be framework agnostic, i.e. we model only resources over time and so are not
limited to map or reduce (or Tez or Reef). We are essentially emulating the
YARN model natively.
* Work purely based on history (can be direct hook to hRaven). We have
developed a Linear Programming model to estimate the resources required for the
job's future execution.
* Optionally reserve the resources using YARN's ReservationSystem.
The gory details are available in the [Morpheus
paper|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].
bq. I would think of this as a natural extension to Timeline Service v2. Do you
think you might want to make use of TSv2 in any way for this?
We can't agree more :). The initial version we have works on YARN RM logs only
because that's what is widely available (in prod deployments). We have designed
it with the clear intent to integrate with TSv2 (or even hRaven) in future,
with the community help, of course.
We will upload a design doc shortly.
> Tool to estimate resource requirements of an application pipeline based on
> prior executions
> -------------------------------------------------------------------------------------------
>
> Key: HADOOP-14840
> URL: https://issues.apache.org/jira/browse/HADOOP-14840
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools
> Reporter: Subru Krishnan
> Assignee: Rui Li
>
> We have been working on providing SLAs for job execution on Hadoop. At high
> level this involves 2 parts: deriving the resource requirements of a job and
> guaranteeing the estimated resources at runtime. The {{YARN
> ReservationSystem}} (YARN-1051/YARN-2572/YARN-5326) enable the latter and in
> this JIRA, we propose to add a tool to Hadoop to predict the resource
> requirements of a job based on past executions of the job. The system (aka
> *Morpheus*) deep dive can be found in our OSDI'16 paper
> [here|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
