I've recently added a bunch of big new machines to my cluster.
However, it seems their defaults for number of files open per user is lower. Though I know I can fix this, I thought the "spilling" code was supposed to prevent running out of open files? Are spills not always closed after they've been written to, or, is there some other growth in the number of open output files? The machines have 1024 files open per user by default, and have been running out of file descriptors (and erroring out the tasks run on them) at spill253.out. For instance: java.io.FileNotFoundException: /state/partition1/hadoop/mapred/local/task_0001_m_000077_0/spill253.out (Too many open files) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(FileInputStream.java:106) at org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.( LocalFileSystem.java:92) at org.apache.hadoop.fs.LocalFileSystem.openRaw(LocalFileSystem.java :143) at org.apache.hadoop.fs.FSDataInputStream$Checker.( FSDataInputStream.java:52) at org.apache.hadoop.fs.FSDataInputStream.(FSDataInputStream.java:279) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:262) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts( MapTask.java:475) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191) at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java :1347) Before anyone asks, yes, my map outputs have *huge* outputs (~1Tb/90 tasks ~= 10Gb each). The problem appears to be the mergeParts job, which looks like it opens up *all* spill files when its merging. Wasn't this code supposed to reduce file-descriptor pressure? Seems like, in this case, if I had only 30 reducers, I'd've been capped at 30 file descriptors per map reduce task. Now, it appears to need outputsize/spillsize file descriptors, which *could* be >> 30. -- Bryan A. P. Pendleton Ph: (877) geek-1-bp
