[
https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800433#comment-13800433
]
Zhijie Shen commented on YARN-925:
----------------------------------
bq. I have two methods in the interface one with the filters and other without
the filters.
It's not about one or two methods. In fact, if we don't supply any filters in
one method, it actually degrades to the other without param. Essentially, we
just need the one with the parameters.
No matter what kind of filters we can provide, one query can still return a
huge number of applications. For example, if we have 1 Million Mapreduce
applications in store, and we filter with applicationType = mapreduce.
What I mean about *two*, is one for small dataset and the other for large
dataset:
* Map<> getAllApplications(...)
* Iterator<> getAllApplications(...)
We need set the upper bound of the number of applications for the first one. On
the other hand, the second one is prepared for iteratively retrieve the
applications one-by-one (maybe not one, but the user defined acceptable
number). Therefore, we don't worry about exploding the memory. The server needs
to implement the detail of how to respond the users one-by-one. Probably it's
not just the scope of HistoryStore, but the end-to-end feature.
bq. Implementation details of these apis, we can discuss in the corrsponding
JIRAS.
I'd like to confirm what filters we want to push back to HistoryStore first,
instead of aggressively adding all. If the implementation of the interface has
no choice but enumerating each applications and then check, the filter is
worthless of being pushed back.
> HistoryStorage Reader Interface for Application History Server
> --------------------------------------------------------------
>
> Key: YARN-925
> URL: https://issues.apache.org/jira/browse/YARN-925
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Fix For: YARN-321
>
> Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch,
> YARN-925-4.patch, YARN-925-5.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
