[
https://issues.apache.org/jira/browse/YARN-5174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327747#comment-15327747
]
Joep Rottinghuis commented on YARN-5174:
----------------------------------------
[~varun_saxena] an image like that does help to explain the hierarchy, and I
think it would be good to include in the documentation.
For the initial verison, do we want to show a flow with different shaped runs?
This is where the concept of flow version comes into play. For automated tuning
tools such as reducer estimation one can compare only two flowruns of the same
"version", meaning that the jobs have the same shape.
I think it might be good for the initial introduction that two flowruns have
the same number of apps and each app the same number of containers.
Similarly, in your flowrun 1, application 2, your attempts 1 and 2 have a
different number of containers. While this is possible through speculative
execution, failed task attempts etc. that is probably not the simplest shape we
can use to introduce the hierarchy.
We could also consider giving the flow and flow run boxes (the vertical
columnar ones) a slightly different color. This would be to indicate two
things:
* Flows and Flow Runs are two new concepts introduced by ATS, and once there
they will hopefully permeate through the rest of Yarn.
* Aggregation up to applications is up to the AM, from then on, ATS v2 will
take care of the aggregation for users.
The documentation can describe that we'll use labels (which go into job configs
if I'm not mistaken) to have users stitch an arbitrary number of applications
into a single flow. This could span MR, Tez, Oozie, Spark, etc. I don't know if
that label / config value work is in yet.
> several updates/corrections to timeline service documentation
> -------------------------------------------------------------
>
> Key: YARN-5174
> URL: https://issues.apache.org/jira/browse/YARN-5174
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Affects Versions: YARN-2928
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Labels: yarn-2928-1st-milestone
> Attachments: Hierarchy.png,
> PublishingApplicationDatatoYARNTimelineServicev.pdf
>
>
> One part that is missing in the documentation is the need to add
> {{hbase-site.xml}} on the client side (the client hadoop cluster). First, we
> need to arrive at the minimally required client setting to connect to the
> right hbase cluster. Then, we need to document it so that users know exactly
> what to do to configure the cluster to use the timeline service v.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
