[
https://issues.apache.org/jira/browse/YARN-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709989#comment-13709989
]
Vinod Kumar Vavilapalli commented on YARN-321:
----------------------------------------------
bq. How about starting with an in-memory implementation, which is the easiest
to do and is useful for testing.
We can have one for testing, but a file-system based impl should exist from the
beginning.
bq. Here's more ellaboration about the per-application data. There should be
three objects to record: RMApp, RMAppAttempt and RMContainer. .....
Tx for the detailed analysis, [~zjshen]!
bq. In addition, HistoryStorage APIs may involve a lot of I/O operations such
that the response of an API will be long. Therefore, it is likely to be good to
make the API non-blocking.
+1
bq. Are we moving aggregated log management(i.e deletion after expiry)
responsibility to AHS?
bq. Right now its not clear what needs to be done for log aggregation?
bq. Sorry for misunderstanding your previous question. IMHO, in the recent
future, we're not moving the aggregated log management, but duplicate it, which
Both AHS and JHS can serve the same aggregated logs. However, AHS and JHS see
the same logs from different point of views. AHS simply considers them as
container logs, no matter what application it is, while JHS know they are the
MR job logs. Vinod Kumar Vavilapalli, would you please confirm it?
That's an interesting point about JHS knowing that they are MR job logs. But we
don't do anything special today. I'm thinking of just pull out log-handling and
use it in both AHS and JHS for the time being.
bq. >>>ResourceManager will push the data to HistoryStorage after an
application finishes in a separate thread.
bq. Is it per application or only one thread in RM?
I foresee a single thread.
bq. Isn't it be a good idea that as soon as application starts we send the
information to AHS and let AHS write all the data published by RM for that
application. In that case it would be very less overhead for RM.
Like I mentioned in my proposal, this could be one implementation of
HistoryStorage.
bq. What about in the cases where RM restart or crashes in those cases RM has
to republish all the running applications to AHS or just forget about the
previous running apps?
This was covered. In the first cut, we'll make best efforts. Losing an app's
data in such a corner case is bearable.
bq. (Thinking out loud) Are we decided on who all can write to HistoryStorage?
I thought this was clear. Only RM. No per-framework data or data from AMs to
AHS directly. Yet, at the least.
bq. I think History storage should be written by AHS not RM in that case RM
will have less load and will be better to scale.
This is an implementation detail. Details on the impl whether the RM writes
files directly or pushes to AHS. Let's do the APIs first.
Overall, I repeat that we should separate the design for per-framework data.
Fundamentally because the AHS is a shared service. It needs a radically
different design from my initial thoughts. Will file a separate JIRA and post
my thoughts there.
> Generic application history service
> -----------------------------------
>
> Key: YARN-321
> URL: https://issues.apache.org/jira/browse/YARN-321
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Luke Lu
> Assignee: Vinod Kumar Vavilapalli
> Attachments: HistoryStorageDemo.java
>
>
> The mapreduce job history server currently needs to be deployed as a trusted
> server in sync with the mapreduce runtime. Every new application would need a
> similar application history server. Having to deploy O(T*V) (where T is
> number of type of application, V is number of version of application) trusted
> servers is clearly not scalable.
> Job history storage handling itself is pretty generic: move the logs and
> history data into a particular directory for later serving. Job history data
> is already stored as json (or binary avro). I propose that we create only one
> trusted application history server, which can have a generic UI (display json
> as a tree of strings) as well. Specific application/version can deploy
> untrusted webapps (a la AMs) to query the application history server and
> interpret the json for its specific UI and/or analytics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
