[
https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13706686#comment-13706686
]
Zhijie Shen commented on YARN-321:
----------------------------------
bq. ResoureManager will write per-application data to a (hopefully very) thin
HistoryStorage layer.
Here's more ellaboration about the per-application data. There should be three
objects to record: RMApp, RMAppAttempt and RMContainer. Bellow are properties
of each object:
Completed Application:
* Application ID
* Application Name
* Application Type
* User
* Queue
* Submit Time
* Start Time
* Finish Time
* Diagnostics Info
* Final Application Status
* Num of Application Attempts
Completed Application Attempt:
* Application Attempt ID
* Application ID
* Host
* RPC Port
* Tracking URL
* Original Tracking URL (not sure it is necessary)
* Diagnostics Info
* Final Application Status
* Master Container ID
* Num of Containers
Completed Container:
* Container ID
* Application Attempt ID
* Final Container Status
* Resource
* Priority
* Node ID
* Log URL
Application has one-to-many relationship with Application Attempt, while
Application Attempt has one-to-one relationship with Container.
WRT the concrete information to record, here's more idea about the interface of
HistoryStorage. The follow APIs should be useful for RM to persist application
history and for AHS to query it:
* Iterable<CompletedApplication> getApplications([conditions...])
* CompletedApplication getApplication(ApplicationId)
* Iterable<CompletedApplicationAttempt> getApplicationAttempts(ApplicationId)
* CompletedApplicationAttempt getApplicationAttempt(ApplicationAttemptId)
* CompletedContainer getContainer(ApplicationAttemptId)
* CompletedContainer getContainer(ContainerId)
* void addApplication(CompletedApplication)
* void addApplicationAttempt(CompletedApplicationAttempt)
* void addContainer(CompletedContainer)
In addition, HistoryStorage APIs may involve a lot of I/O operations such that
the response of an API will be long. Therefore, it is likely to be good to make
the API non-blocking.
> Generic application history service
> -----------------------------------
>
> Key: YARN-321
> URL: https://issues.apache.org/jira/browse/YARN-321
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Luke Lu
> Assignee: Vinod Kumar Vavilapalli
>
> The mapreduce job history server currently needs to be deployed as a trusted
> server in sync with the mapreduce runtime. Every new application would need a
> similar application history server. Having to deploy O(T*V) (where T is
> number of type of application, V is number of version of application) trusted
> servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and
> history data into a particular directory for later serving. Job history data
> is already stored as json (or binary avro). I propose that we create only one
> trusted application history server, which can have a generic UI (display json
> as a tree of strings) as well. Specific application/version can deploy
> untrusted webapps (a la AMs) to query the application history server and
> interpret the json for its specific UI and/or analytics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
