[
https://issues.apache.org/jira/browse/YARN-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609821#comment-16609821
]
Jason Lowe commented on YARN-8680:
----------------------------------
Thanks for updating the patch!
In loadUserLocalizedResources for this patch hunk:
{noformat}
- if (!key.startsWith(keyPrefix)) {
+
+ if (!key.startsWith(LOCALIZATION_APPCACHE_SUFFIX,
+ keyPrefix.length())) {
break;
}
{noformat}
The old code would make sure the key matches the expected prefix, but the new
code is making the dangerous assumption that the key found has the same base
prefix that was used in the seek. That is not necessarily the case. If there
are no appcache localization entries in the database then this will seek to the
first key that occurs lexicographically _after_ the desired key. That key may
or may not be long enough to seek keyPrefix.length() characters into it, and if
it isn't then we explode with an index out of bounds exception. This code
needs to walk through a sub-block of keys by checking the full key prefix break
out of the loop when it doesn't match. Just above the while loop the code
computes the desired prefix, so it just needs to cache it in a local variable
for later comparison in the while loop.
Same comment applies to the handling of the LOCALIZATION_FILECACHE_SUFFIX key
after the while loop.
> YARN NM: Implement Iterable Abstraction for LocalResourceTrackerstate
> ---------------------------------------------------------------------
>
> Key: YARN-8680
> URL: https://issues.apache.org/jira/browse/YARN-8680
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Reporter: Pradeep Ambati
> Assignee: Pradeep Ambati
> Priority: Critical
> Attachments: YARN-8680.00.patch, YARN-8680.01.patch,
> YARN-8680.02.patch, YARN-8680.03.patch
>
>
> Similar to YARN-8242, implement iterable abstraction for
> LocalResourceTrackerState to load completed and in progress resources when
> needed rather than loading them all at a time for a respective state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
