[
https://issues.apache.org/jira/browse/HDFS-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855012#action_12855012
]
Tsz Wo (Nicholas), SZE commented on HDFS-1085:
----------------------------------------------
{code}
//org.apache.hadoop.hdfs.server.namenode.StreamFile.doGet(..)
try {
int bytesRead;
while ((bytesRead = in.read(buf)) != -1) {
os.write(buf, 0, bytesRead);
}
} finally {
in.close();
os.close();
dfs.close();
}
{code}
In the codes above, if in.read(..) or os.write(..) throw an exception (checked
or unchecked), the streams will be closed as usual. Then, client side won't be
able to know about the exception but considers that it is an EOF.
> hftp read failing silently
> ---------------------------
>
> Key: HDFS-1085
> URL: https://issues.apache.org/jira/browse/HDFS-1085
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Koji Noguchi
>
> When performing a massive distcp through hftp, we saw many tasks fail with
> {quote}
> 2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL
> 2010/0/part-00032 : java.io.IOException: File size not matched: copied
> 193855488 bytes (184.9m) to tmpfile
> (=hdfs://omehost.com:8020/somepath/part-00032)
> but expected 1710327403 bytes (1.6g) from
> hftp://someotherhost/somepath/part-00032
> at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
> at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
> at org.apache.hadoop.mapred.Child.main(Child.java:159)
> {quote}
> This means that read itself didn't fail but the resulted file was somehow
> smaller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
