We got the same problem as you when using MultipleOutputFormat both on
hadoop 0.18 and 0.19. On hadoop 0.18, increasing the xceivers count does
not fix the problem. But we found many error message complaining that
xceiverCount exceeded the limit of concurrent xcievers in datanode
(running on hadoop 0.19) log. After we increased the xceivers count, the
problem was gone.
I guess you are using hadoop 0.18. Please try 0.19.
Good luck.
Scott Whitecross wrote:
I tried modifying the settings, and I'm still running into the same
issue. I increased the xceivers count (fs.datanode.max.xcievers) in
the hadoop-site.xml file. I also checked to make sure the file
handles were increased, but they were fairly high to begin with.
I don't think I'm dealing with anything out of the ordinary either.
I'm process three large 'log' files, totaling around 5 GB, and
producing around 8000 output files after some data processing,
probably totals 6 or 7 gig. In the past, I've produced a lot fewer
files, and that has been fine. When I change the process to output to
just a few files, no problem again.
Anything else beyond the limits? Is HDFS creating a substantial
amount of temp files as well?
On Feb 9, 2009, at 8:11 PM, Bryan Duxbury wrote:
Correct.
+1 to Jason's more unix file handles suggestion. That's a must-have.
-Bryan
On Feb 9, 2009, at 3:09 PM, Scott Whitecross wrote:
This would be an addition to the hadoop-site.xml file, to up
dfs.datanode.max.xcievers?
Thanks.
On Feb 9, 2009, at 5:54 PM, Bryan Duxbury wrote:
Small files are bad for hadoop. You should avoid keeping a lot of
small files if possible.
That said, that error is something I've seen a lot. It usually
happens when the number of xcievers hasn't been adjusted upwards
from the default of 256. We run with 8000 xcievers, and that seems
to solve our problems. I think that if you have a lot of open
files, this problem happens a lot faster.
-Bryan
On Feb 9, 2009, at 1:01 PM, Scott Whitecross wrote:
Hi all -
I've been running into this error the past few days:
java.io.IOException: Could not get block locations. Aborting...
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)
It seems to be related to trying to write to many files to HDFS.
I have a class extending
org.apache.hadoop.mapred.lib.MultipleOutputFormat and if I output
to a few file names, everything works. However, if I output to
thousands of small files, the above error occurs. I'm having
trouble isolating the problem, as the problem doesn't occur in the
debugger unfortunately.
Is this a memory issue, or is there an upper limit to the number
of files HDFS can hold? Any settings to adjust?
Thanks.
