Re: mapreduce memory issues

[Tamas Jambor]

oh ok. this is the problem. I set mapred.child.java.opts to -Xmx1024m, 
but that doesn't match the regex expression, so it sets io.sort.mb to 
1000. Changing it to -Xmx([0-9]+)m solved the problem.

On 05/05/2010 22:26, Sean Owen wrote:
You are welcome to test this by modifying setIOSort() in
RecommenderJob. Just crank down that "1000", or divide heapMB by a
larger value. This should let you run with somewhat smaller heap
sizes. If that's the culprit I'm happy to turn down these values a
bit.