[
https://issues.apache.org/jira/browse/YARN-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592706#comment-16592706
]
Pradeep Ambati commented on YARN-8680:
--------------------------------------
Main idea in the fix was taken from [~jlowe] comment (added below) in
[-YARN-8242-|https://issues.apache.org/jira/browse/YARN-8242].
{quote}Rather than a full re-iteration, we can seek to a key that we know is
after a user's localization entries but necessarily before any other user's
entry. Seeking is very fast and done all the time during recovery, so it would
be much faster than iterating. For example, userA's private localization
entries will have a key prefix of "Localization/private/userA/" and have
entries with a prefix of either "Localization/private/userA/filecache/" or
"Localization/private/userA/appcache/". If we seek to a key that occurs
lexicographically after those prefixes, like "Localization/private/userA/zzz",
then we will have an iterator starting after the localization records for userA
but necessarily before any user that occurs after userA lexicographically. That
avoids the double-iteration performance problem nor does it rely on approaches
that would require the previous user iterator to be fully consumed to function
properly.
{quote}
The seeking mechanism pointed in the above comment is used for seeking multiple
users as well as multiple applications of each respective users.
> YARN NM: Implement Iterable Abstraction for LocalResourceTrackerstate
> ---------------------------------------------------------------------
>
> Key: YARN-8680
> URL: https://issues.apache.org/jira/browse/YARN-8680
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Reporter: Pradeep Ambati
> Assignee: Pradeep Ambati
> Priority: Critical
> Attachments: YARN-8680.00.patch
>
>
> Similar to YARN-8242, implement iterable abstraction for
> LocalResourceTrackerState to load completed and in progress resources when
> needed rather than loading them all at a time for a respective state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
