Rui Shi wrote:
It is hard to believe that you need to enlarge heap size given the input size is only 10MB. In particular, you don't load all input at the same time. As for the program logic, no much fancy stuff, mostly cut and sorting. So GC should be able to handle...
Out-of-memory exceptions can also be caused by having too many files open at once. What does 'ulimit -n' show?
You presented an excerpt from a jobtracker log, right? What do the tasktracker logs show?
Can you monitor a node while it is running to see whether the jvm's heap is growing, or whether the number of open files (lsof -p) is large?
Also, can you please provide more details about your application? I.e., what is your inputformat, map function, etc.
Doug
