[ https://issues.apache.org/jira/browse/YARN-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838851#comment-15838851 ]
Chris Trezzo commented on YARN-3637: ------------------------------------ Thanks [~sjlee0] for the commit/review and thanks [~templedf] for the review! > Handle localization sym-linking correctly at the YARN level > ----------------------------------------------------------- > > Key: YARN-3637 > URL: https://issues.apache.org/jira/browse/YARN-3637 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Chris Trezzo > Assignee: Chris Trezzo > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-3637-trunk.001.patch, YARN-3637-trunk.002.patch, > YARN-3637-trunk.003.patch > > > The shared cache needs to handle resource sym-linking at the YARN layer. > Currently, we let the application layer (i.e. mapreduce) handle this, but it > is probably better for all applications if it is handled transparently. > Here is the scenario: > Imagine two separate jars (with unique checksums) that have the same name > job.jar. > They are stored in the shared cache as two separate resources: > checksum1/job.jar > checksum2/job.jar > A new application tries to use both of these resources, but internally refers > to them as different names: > foo.jar maps to checksum1 > bar.jar maps to checksum2 > When the shared cache returns the path to the resources, both resources are > named the same (i.e. job.jar). Because of this, when the resources are > localized one of them clobbers the other. This is because both symlinks in > the container_id directory are the same name (i.e. job.jar) even though they > point to two separate resource directories. > Originally we tackled this in the MapReduce client by using the fragment > portion of the resource url. This, however, seems like something that should > be solved at the YARN layer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org