Hi All, While reading input from directory recursively consisting of files of size 30Mb, using WholeFileInputFormat and WholeFileRecordReader, I am running into JavaHeapSize error for even a very small file of 30MB. By default the mapred.child.java.opts is set to -Xmx200m and should be sufficient enough to run atleast 30MB files present in the directory.
The input is a normal random words in file. Each Map is given a single file of size 30MB and I am reading value as the content of the whole file. And running normal word count. If I increase the mapred.child.java.opts size to higher value the applications runs successfully. But it would be great if anyone can suggest me why mapred.child.java.opts which is currently 200Mb default for task is not sufficient for 30 MB file, as this means Hadoop MapReduce is consuming a lot of heap size and out of 200MB it doesn't even use 30Mb to process the task? Also, is there any other way to read the a large Whole file as a input to a single Map, meaning every Map gets a whole file to process? -Shubh
