Oh duh, I was calling Matcher.matches() instead of Matcher.find(), which caused it to only match the whole string. I can fix that and improve this regex, and make the default buffer smaller.
Where do you see it setting mappers and reducers? it shouldn't, and doesn't as far as I can see. Hadoop may be overriding your settings. On Wed, May 5, 2010 at 11:11 PM, Tamas Jambor <jambo...@googlemail.com> wrote: > oh ok. this is the problem. I set mapred.child.java.opts to -Xmx1024m, but > that doesn't match the regex expression, so it sets io.sort.mb to 1000. > Changing it to -Xmx([0-9]+)m solved the problem. > > On 05/05/2010 22:26, Sean Owen wrote: >> >> You are welcome to test this by modifying setIOSort() in >> RecommenderJob. Just crank down that "1000", or divide heapMB by a >> larger value. This should let you run with somewhat smaller heap >> sizes. If that's the culprit I'm happy to turn down these values a >> bit. >> >
