[
https://issues.apache.org/jira/browse/MAPREDUCE-5960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-5960:
-------------------------------------
Attachment: MAPREDUCE-5960.v02.patch
[~jlowe], thanks for looking into this JIRA.
I agree that it might not be safe to assume that the remote source be
accessible from the server side. However, the user will find this out soon
enough because the job will fail. It's equally wrong to assume the job jar
won't be accessible, and deprive users from the option to store job bits at
some other place central, such as s3. We can argue that it would be more
efficient to first automatically copy the source jar to staging dir, and have
NM localize from there as opposed to the original source.
Setting this discussion aside, it does not look good that the submission simply
fails. I added a unit test for the scenario I believe just has to work
functionally.
> JobSubmitter's check whether job.jar is local is incorrect with no authority
> in job jar path.
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5960
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5960
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.4.0, 2.4.1
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5960.v01.patch, MAPREDUCE-5960.v02.patch
>
>
> {code}
> $ mapred job -submit myjob.xml
> 2014-07-06 15:06:34.731 java[12120:1903] Unable to load realm info from
> SCDynamicStore
> 14/07/06 15:06:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/07/06 15:06:34 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/07/06 15:06:37 WARN conf.Configuration: bad conf file: element not
> <property>
> 14/07/06 15:06:37 WARN conf.Configuration: bad conf file: element not
> <property>
> 14/07/06 15:06:37 INFO client.RMProxy: Connecting to ResourceManager at
> /0.0.0.0:8032
> 14/07/06 15:11:11 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/gshegalov/.staging/job_1404679996131_0004
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
> viewfs:/user/gshegalov/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar,
> expected: file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:80)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:525)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:739)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1902)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1870)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1835)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyJar(JobSubmitter.java:286)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:254)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:260)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1239)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
