wangmeng created HDFS-5996:
------------------------------
Summary: hadoop 1.12. hdfs write bug
Key: HDFS-5996
URL: https://issues.apache.org/jira/browse/HDFS-5996
Project: Hadoop HDFS
Issue Type: Bug
Components: fuse-dfs
Affects Versions: 1.1.2
Environment: one master and three slave ,all of them are normal
Reporter: wangmeng
Fix For: 1.1.2
I am a student from China ,my research is Hive data storage on hadoop .There
is a hdfs-write bug when I used sql : insert overwrite table wangmeng
select * from testTable (this sql is translated into N map( no Reduce)
jobs,each map .corresponding to a HDFS file output On disk. ) No matter
what value N is , there will always exists some DfsdataoutputStream buffer
can not write to disk at last ,such as N=160 files ,then there my be
about 5 write-faliure files .,the write-failured hdfs--file size on disk
is always 0 bytes rather than a value which is between 0 and zhe correct
size. .There does not have any exceptions to throw . and the HDFS WRITTEN
statisticcal data is absolutely correct . When I debug , I find those
write-failed DFS-buffer own absolutely correct values on its buffer ,but
the buffer can not write to disk at last although I use
Dfsdataoutputstream.flush , Dfsdataoutputstream close . .I can not find the
reason those dfs-buffer can not write success. Now I choose a method to
avoide this problem --use a temporary file : for example , if the
DFS-buffer will write to its destination FINAL, now I will let this DFS-buffer
write to a temporary file TEM first ,and then I move the TEM data to
the destination just by change the hdfs-- file path. This method can avoid
the DFS-buffer write -failure .Now I want to fix this problem
radically , so How can I patch my codes about this problem and is there
anything I can do ? Many Thanks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
