I've recently added a bunch of big new machines to my cluster.

However, it seems their defaults for number of files open per user is lower.
Though I know I can fix this, I thought the "spilling" code was supposed to
prevent running out of open files? Are spills not always closed after
they've been written to, or, is there some other growth in the number of
open output files?

The machines have 1024 files open per user by default, and have been running
out of file descriptors (and erroring out the tasks run on them) at
spill253.out. For instance:

java.io.FileNotFoundException:
/state/partition1/hadoop/mapred/local/task_0001_m_000077_0/spill253.out (Too
many open files)
   at java.io.FileInputStream.open(Native Method)
   at java.io.FileInputStream.(FileInputStream.java:106)
   at org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.(
LocalFileSystem.java:92)
   at org.apache.hadoop.fs.LocalFileSystem.openRaw(LocalFileSystem.java
:143)
   at org.apache.hadoop.fs.FSDataInputStream$Checker.(
FSDataInputStream.java:52)
   at org.apache.hadoop.fs.FSDataInputStream.(FSDataInputStream.java:279)
   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:262)
   at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(
MapTask.java:475)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
   at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
:1347)


Before anyone asks, yes, my map outputs have *huge* outputs (~1Tb/90 tasks
~= 10Gb each).

The problem appears to be the mergeParts job, which looks like it opens up
*all* spill files when its merging. Wasn't this code supposed to reduce
file-descriptor pressure? Seems like, in this case, if I had only 30
reducers, I'd've been capped at 30 file descriptors per map reduce task.
Now, it appears to need outputsize/spillsize file descriptors, which *could*
be >> 30.

--
Bryan A. P. Pendleton
Ph: (877) geek-1-bp

Reply via email to