[
https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709030#comment-13709030
]
Mayank Bansal commented on YARN-321:
------------------------------------
Overall Looks good, However some points to consider
>>>ResourceManager will push the data to HistoryStorage after an application
>>>finishes in a separate thread.
Is it per application or only one thread in RM?
Isn't it be a good idea that as soon as application starts we send the
information to AHS and let AHS write all the data published by RM for that
application. In that case it would be very less overhead for RM.
What about in the cases where RM restart or crashes in those cases RM has to
republish all the running applications to AHS or just forget about the previous
running apps?
Right now its not clear what needs to be done for log aggregation?
> Generic application history service
> -----------------------------------
>
> Key: YARN-321
> URL: https://issues.apache.org/jira/browse/YARN-321
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Luke Lu
> Assignee: Vinod Kumar Vavilapalli
>
> The mapreduce job history server currently needs to be deployed as a trusted
> server in sync with the mapreduce runtime. Every new application would need a
> similar application history server. Having to deploy O(T*V) (where T is
> number of type of application, V is number of version of application) trusted
> servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and
> history data into a particular directory for later serving. Job history data
> is already stored as json (or binary avro). I propose that we create only one
> trusted application history server, which can have a generic UI (display json
> as a tree of strings) as well. Specific application/version can deploy
> untrusted webapps (a la AMs) to query the application history server and
> interpret the json for its specific UI and/or analytics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
