[
https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated YARN-953:
-----------------------------
Attachment: YARN-953.8.patch
bq. history-writer.dispatcher-vector-size ->
history-writer.multithreadeddispatcher.pool-size.
Done
bq. history-writer.store.class -> history-writer.class
Done
bq. I thought DummyApplicationHistoryStore is going away. But it's fine I
guess. Atleast DummyApplicationHistoryStore -> NullApplicationHistoryStore.
Done
bq. No need to mention version numbers in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml?
The version seems to be required here. I think the reason may be the maven meta
data for this sub-project is not now
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop
Anyway, I changed it to:
{code}
+ <version>${project.version}</version>
{code}
Then, we don't need to worry about it when merge it to trunk.
bq. Change the dispatcher name to be something like MultiThreadedDispatcher ?
Done
bq. WritingHistoryEventType: Drop WRITING_ prefix from all events.
Instead of putting in more code, let's fix the app-finishes-before-appattempt
bug when killing an app separately. We discussed this in the context of RM
restart too.
Done
bq. Instead of creating new events from container, we can write
container-history events in app when RMAppAttemptContainerAllocatedEvent and
RMAppAttemptContainerFinishedEvent. Otherwise, we risk creating far more events.
In the last patch, I've already avoided adding new events.
bq. TestRMApplicationHistoryWriter: increment -> incrementCounts
Done
In addition, I've added a stress test, and also a demo that RM can run with
writing history data. In the test, I cannot independently evaluate the how fast
of processing writing history events, because their many other events need to
be processed as well for MockRM. I did a trick by comparing the time needed
with and without processing history events. The difference is usually tiny.
> [YARN-321] Enable ResourceManager to write history data
> -------------------------------------------------------
>
> Key: YARN-953
> URL: https://issues.apache.org/jira/browse/YARN-953
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Zhijie Shen
> Attachments: YARN-953-5.patch, YARN-953.1.patch, YARN-953.2.patch,
> YARN-953.3.patch, YARN-953.4.patch, YARN-953.5.patch, YARN-953.6.patch,
> YARN-953.7.patch, YARN-953.8.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
