[
https://issues.apache.org/jira/browse/HDFS-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637783#comment-13637783
]
Jerry He commented on HDFS-4720:
--------------------------------
Is this the message and stack before your fix?
$ hadoop distcp /tmp/jruby-complete-1.6.7.jar webhdfs://hdtest006:14000/tmp
13/04/21 21:15:18 INFO tools.DistCp: srcPaths=[/tmp/jruby-complete-1.6.7.jar]
13/04/21 21:15:18 INFO tools.DistCp: destPath=webhdfs://hdtest006:14000/tmp
13/04/21 21:15:19 INFO tools.DistCp: sourcePathsCount=1
13/04/21 21:15:19 INFO tools.DistCp: filesToCopyCount=1
13/04/21 21:15:19 INFO tools.DistCp: bytesToCopyCount=13.0m
13/04/21 21:15:19 WARN web.WebHdfsFileSystem: Original exception is
org.apache.hadoop.ipc.RemoteException: File does not exist:
/tmp/_distcp_logs_dmfa5b
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:294)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:103)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:547)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:471)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:404)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:568)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:579)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
at
org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:120)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:951)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at
java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:886)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
13/04/21 21:15:19 INFO mapred.JobClient: Running job: job_201304191106_0012
13/04/21 21:15:20 INFO mapred.JobClient: map 0% reduce 0%
13/04/21 21:15:28 INFO mapred.JobClient: map 100% reduce 0%
13/04/21 21:15:29 INFO mapred.JobClient: Job complete: job_201304191106_0012
13/04/21 21:15:29 INFO mapred.JobClient: Counters: 22
13/04/21 21:15:29 INFO mapred.JobClient: Job Counters
13/04/21 21:15:29 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=6290
13/04/21 21:15:29 INFO mapred.JobClient: Launched map tasks=1
13/04/21 21:15:29 INFO mapred.JobClient: Total time spent by all reduces
waiting after reserving slots (ms)=0
13/04/21 21:15:29 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
13/04/21 21:15:29 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/04/21 21:15:29 INFO mapred.JobClient: distcp
13/04/21 21:15:29 INFO mapred.JobClient: Bytes expected=13660468
13/04/21 21:15:29 INFO mapred.JobClient: Files copied=1
13/04/21 21:15:29 INFO mapred.JobClient: Bytes copied=13660468
13/04/21 21:15:29 INFO mapred.JobClient: FileSystemCounters
13/04/21 21:15:29 INFO mapred.JobClient: HDFS_BYTES_READ=13660867
13/04/21 21:15:29 INFO mapred.JobClient: FILE_BYTES_WRITTEN=25858
13/04/21 21:15:29 INFO mapred.JobClient: WEBHDFS_BYTES_WRITTEN=13660468
13/04/21 21:15:29 INFO mapred.JobClient: File Output Format Counters
13/04/21 21:15:29 INFO mapred.JobClient: Bytes Written=0
13/04/21 21:15:29 INFO mapred.JobClient: Map-Reduce Framework
13/04/21 21:15:29 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=898535424
13/04/21 21:15:29 INFO mapred.JobClient: Map input bytes=160
13/04/21 21:15:29 INFO mapred.JobClient: Physical memory (bytes)
snapshot=69738496
13/04/21 21:15:29 INFO mapred.JobClient: Map output records=0
13/04/21 21:15:29 INFO mapred.JobClient: CPU time spent (ms)=590
13/04/21 21:15:29 INFO mapred.JobClient: Map input records=1
13/04/21 21:15:29 INFO mapred.JobClient: Total committed heap usage
(bytes)=5669888
13/04/21 21:15:29 INFO mapred.JobClient: Spilled Records=0
13/04/21 21:15:29 INFO mapred.JobClient: SPLIT_RAW_BYTES=139
13/04/21 21:15:29 INFO mapred.JobClient: File Input Format Counters
13/04/21 21:15:29 INFO mapred.JobClient: Bytes Read=260
> Misleading warning message in WebhdfsFileSystem when trying to check whether
> path exist using webhdfs url
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-4720
> URL: https://issues.apache.org/jira/browse/HDFS-4720
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 1.1.1, 1.1.2
> Reporter: Yu Li
> Assignee: Yu Li
> Priority: Minor
> Attachments: HDFS-4720-trunk.patch
>
>
> When we trying to check whether the target path exists in HDFS through
> webhdfs, if the given path to check doesn't exist, we will always observe
> warning message like:
> ===========================================================
> 13/04/21 04:38:01 WARN web.WebHdfsFileSystem: Original exception is
> org.apache.hadoop.ipc.RemoteException: user = biadmin, proxyUser = null, path
> = /testWebhdfs
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:294)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:103)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:552)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:473)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:404)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:573)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:584)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> ===========================================================
> while actually FileNotFoundException should be expected when the operation is
> GETFILESTATUS and target path doesn't exist. The fact that RemoteException
> didn't include the real exception class(FileNotFoundException) in its
> toString method even make the message more misleading, since from the message
> user won't know what the warning is about
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
