[
https://issues.apache.org/jira/browse/YARN-5306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362697#comment-15362697
]
Junping Du commented on YARN-5306:
----------------------------------
Thanks [~yeshavora] for reporting the issue here. Localize the same resource
with different timestamp (if it is unexpected) is a very tricky issue and is
hard to debug for now as you cannot differentiate this case from the other case
that the source file is actually changed during localization process.
If the fix is to detect and fail fast for duplicated resources case, then it
shouldn't be hard as we can just add a map from resource path to timestamp
which can complain earlier that if another resource with the same path but
different timestamp. My only concern here is if we should support the case that
resource override - assume we have common used bundle of jars (in format of tar
ball, etc.) in distributed cache and we want to replace one of them in one job
run. However, this sounds like not work today as the complain on timestamp will
happen during localization but I would like to hear ideas or real use cases
here.
> Yarn should detect and fail fast on duplicate resources in container request
> ----------------------------------------------------------------------------
>
> Key: YARN-5306
> URL: https://issues.apache.org/jira/browse/YARN-5306
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: Yesha Vora
> Assignee: Junping Du
> Priority: Critical
>
> In some cases, Yarn gets duplicate copies of resources in resource-list.
> In this case, you end up with a resource list which contains two copies of
> resource JAR, with the timestamps of the two separate uploads —only one of
> which (the later one) is correct. At download time, the NM goes through the
> list and fails the download when it gets to the one with the older timestamp.
> We need some utility class to do a scan & check could be used by the NM at
> download time (so fail with meaningful errors), and the yarn client could
> perhaps do the check before launch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
