[
https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853492#comment-13853492
]
Zhijie Shen commented on YARN-321:
----------------------------------
AFAIK,
bq. Will the AHS serve logs?
It will serve the aggregated logs.
bq. Is my understanding correct that for the first version, the RM will not do
any RPCs to the AHS.
Yes, RM and AHS don't communicate via RPC, but the shared bus.
bq. Will APIs be Public / Stable?
Assume you're talking about RPC interface. It will be Public but Unstable.
bq. In the first version, what will be provided in terms of web UI?
* The page of a list of completed applications (FINISHED, FAILED, KILLED) in
HistoryStorage
* The page of one application and a list of its attempts
* The page of one attempt and a list of its containers
* The page of one container
* The page of the container logs
bq. How are ACLs enforced between the AHS and RM?
Is your question how the ACLs enforced on one application when it is running by
RM is inherited by AHS when the application is finished and stored in
HistoryStorage? My rough idea is to persist the ACLs info into HistoryStorage
as well. For ApplicationACLs, it seems to be much obvious. For QueueACLs, I'm
not quite sure, may be complex and depend on scheduler.
bq. Eventually we want the AHS to interact well with long-running services,
right?
I think so.
bq. For the file-based bus, have we though about an approach for how it might
store info for applications in non-terminal states?
Currently, HistoryStorage is receiving a series of events which notifies the
updates about an application/attempt/container. The info is accumulated
incrementally. IMO, the strategy can be applied to a long-running app with
defining more intermediate events (now we only defines start and finish events)
to report the progress. Then, instead of holding the events until app gets
finished, we need to flush them in advance/periodically/..., such that users
can get informed of latest progress of the app.
Hopefully the answers can address you questions.:-)
> Generic application history service
> -----------------------------------
>
> Key: YARN-321
> URL: https://issues.apache.org/jira/browse/YARN-321
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Luke Lu
> Assignee: Vinod Kumar Vavilapalli
> Attachments: AHS Diagram.pdf, ApplicationHistoryServiceHighLevel.pdf,
> Generic Application History - Design-20131219.pdf, HistoryStorageDemo.java
>
>
> The mapreduce job history server currently needs to be deployed as a trusted
> server in sync with the mapreduce runtime. Every new application would need a
> similar application history server. Having to deploy O(T*V) (where T is
> number of type of application, V is number of version of application) trusted
> servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and
> history data into a particular directory for later serving. Job history data
> is already stored as json (or binary avro). I propose that we create only one
> trusted application history server, which can have a generic UI (display json
> as a tree of strings) as well. Specific application/version can deploy
> untrusted webapps (a la AMs) to query the application history server and
> interpret the json for its specific UI and/or analytics.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
