[
https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13856021#comment-13856021
]
Vinod Kumar Vavilapalli commented on YARN-321:
----------------------------------------------
Tx Zhijie for answering most of Sandy's questions, you are spot on. I'll update
the design doc to clarify things where it isn't clear.
bq. What is the jira for app specific history data?
I just filed YARN-1530, will post more information soon.
bq. Could you describe the security requirements a bit further. Its not clear
to everyone how everything works currently. To be clear, what exactly needs to
be done to make apps write and read history data.
The data covered in this JIRA is generic and only RM gets to write it. The
consumers of this data are *both* the cluster admins for historical analyses as
well as individual apps that chose to not use features that come out of
YARN-1530. As such, we cannot let apps write history data.
bq. How is the shared bus different from writing to a file. I would think one
would cover the other.
Yes, writing to a file is one example of a shared-bus. I'll fix it if the doc
is confusing w.r.t this.
> Generic application history service
> -----------------------------------
>
> Key: YARN-321
> URL: https://issues.apache.org/jira/browse/YARN-321
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Luke Lu
> Assignee: Vinod Kumar Vavilapalli
> Attachments: AHS Diagram.pdf, ApplicationHistoryServiceHighLevel.pdf,
> Generic Application History - Design-20131219.pdf, HistoryStorageDemo.java
>
>
> The mapreduce job history server currently needs to be deployed as a trusted
> server in sync with the mapreduce runtime. Every new application would need a
> similar application history server. Having to deploy O(T*V) (where T is
> number of type of application, V is number of version of application) trusted
> servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and
> history data into a particular directory for later serving. Job history data
> is already stored as json (or binary avro). I propose that we create only one
> trusted application history server, which can have a generic UI (display json
> as a tree of strings) as well. Specific application/version can deploy
> untrusted webapps (a la AMs) to query the application history server and
> interpret the json for its specific UI and/or analytics.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
