[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093050#comment-14093050
]
Wing Yew Poon commented on MAPREDUCE-6027:
------------------------------------------
Karthik,
in ClientDistributedCacheManager.java before my change. there was a
getFileStatus static method with signature getFileStatus(Configuration, URI,
Map<URI, FileStatus>). It was called in one method, determineTimestamps (where
it's called 4 times) and it called getFileStatus(FileSystem, URI, Map<URI,
FileStatus>) to do the work. In addition, getFileStatus(FileSystem, URI,
Map<URI, FileStatus>) was also called by checkPermissionOfOther (once).
In determineTimestamps, there are only URIs, not Paths. So it is not natural to
convert the URIs to Paths (by doing new Path(uri)) and then call a
getFileStatus method with a Path parameter, where it then converts the Path
back to a URI (path.toUri()) before looking it up in the cache. In
checkPermissionOfOther, however, there is a Path to begin with, hence it made
sense to call a getFileStatus method with a Path parameter.
I thought hard about how to make my change without duplicating code, but I
didn't see any easy way to do it.
I can implement your suggestion, so that getFileStatus(Configuration, URI,
Map<URI, FileStatus>) calls getFileStatus(FileSystem, Path, Map<URI,
FileStatus>) - getting rid of getFileStatus(FileSystem, URI, Map<URI,
FileStatus>) - but that means converting URIs to Paths and then back to URIs
for the cache lookup.
As to your second point, in practice, the way the project is built (from
maven), TEST_ROOT_DIR is always CURRENT_DIR + "target/test-dir/". So,
secondCacheFile is always a relative Path,. In case the test is run
differently, e.g., the system property test.build.data is not set by the build
(so defaults to /tmp) or set to something else, a valid secondCacheFile is
still constructed, and both the tests are still valid; they just won't test a
relative Path. I didn't want to just do
{noformat}
secondCacheFile = new Path("secondcachefile");
{noformat}
as I didn't want to pollute the project directory.
> mr jobs with relative paths can fail
> ------------------------------------
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
> Reporter: Wing Yew Poon
> Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 100000 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {noformat}
> If I used absolute paths for the input and out directories, the job runs fine.
> This breakage is due to HADOOP-10876.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
