[
https://issues.apache.org/jira/browse/YARN-956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated YARN-956:
-----------------------------
Attachment: YARN-956.4.patch
I uploaded a new patch, which completely rewrites the
MemoryApplicationHistoryStore. The major changes are listed bellow:
1. Make MemoryApplicationHistoryStore extends AbstractService, and make the
constructor public, as RM can uniformly use Java reflection to construct any
implementation of ApplicationHistoryStore.
2. Make the in-memory store the static objects, thus it will not be erased when
MemoryApplicationHistoryStore instance is destroyed. RM can create one instance
to use the writer interface to add history data, while AHS can also create
another instance to use the reader interface to read history data. The static
references were the common place where all instances can access, like the root
dir in the file system for FileSystemApplicationHistoryStore.
3. Make containerData have the two-level lookup index as well, because multiple
containers may belongs to one application attempt. It can speed up the reader
interface: Map<ContainerId, ContainerHistoryData>
getContainers(ApplicationAttemptId appAttemptId)
4. Keep storing XXXXHistoryData instead of storing XXXXStartData and
XXXXFinishData. When XXXXStartData is to be written, XXXXHistoryData will be
created, and copy the fields in XXXXStartData into those in XXXXHistoryData.
When XXXXFinishData is to be written, merge fields in XXXXFinishData into the
existing XXXXHistoryData. This helps to simplify the implementation of the
reader interface and improve its performance.
5. Set the rule of not overriding the existing history data, because the each
record is supposed to be unique and just written once. An IOException will be
thrown if it happens. In addition, XXXXStartData is supposed to be written
before XXXXFinishData. Otherwise, an IOException will be thrown as well.
6. Rewrite the test cases to verify both the correct read/write procedures, and
the incorrect writing operations.
> [YARN-321] Add a testable in-memory HistoryStorage
> ---------------------------------------------------
>
> Key: YARN-956
> URL: https://issues.apache.org/jira/browse/YARN-956
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Zhijie Shen
> Fix For: YARN-321
>
> Attachments: YARN-956-1.patch, YARN-956-2.patch, YARN-956-3.patch,
> YARN-956.4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
