[
https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853439#comment-13853439
]
Sandy Ryza commented on YARN-321:
---------------------------------
Thanks Vinod for the design doc. A few additional questions:
Will the AHS serve logs?
Is my understanding correct that for the first version, the RM will not do any
RPCs to the AHS.
Will APIs be Public / Stable?
In the first version, what will be provided in terms of web UI?
How are ACLs enforced between the AHS and RM?
Eventually we want the AHS to interact well with long-running services, right?
For the file-based bus, have we though about an approach for how it might store
info for applications in non-terminal states?
> Generic application history service
> -----------------------------------
>
> Key: YARN-321
> URL: https://issues.apache.org/jira/browse/YARN-321
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Luke Lu
> Assignee: Vinod Kumar Vavilapalli
> Attachments: AHS Diagram.pdf, ApplicationHistoryServiceHighLevel.pdf,
> Generic Application History - Design-20131219.pdf, HistoryStorageDemo.java
>
>
> The mapreduce job history server currently needs to be deployed as a trusted
> server in sync with the mapreduce runtime. Every new application would need a
> similar application history server. Having to deploy O(T*V) (where T is
> number of type of application, V is number of version of application) trusted
> servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and
> history data into a particular directory for later serving. Job history data
> is already stored as json (or binary avro). I propose that we create only one
> trusted application history server, which can have a generic UI (display json
> as a tree of strings) as well. Specific application/version can deploy
> untrusted webapps (a la AMs) to query the application history server and
> interpret the json for its specific UI and/or analytics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
