RecommenderJob and io.sort.mb

Sun, 18 Sep 2011 06:26:10 -0700 

I'm trying to run the RecommenderJob (trunk as of this morning) and am getting:
java.io.IOException: Invalid "io.sort.mb": 2048
        at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:939)
        at 
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:673)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.hadoop.mapred.Child.main(Child.java:253)


My heap size is 4gb.  AFAICT, the issue is in the RecommenderJob, line 270.   
The problem is due to https://issues.apache.org/jira/browse/MAPREDUCE-2308

Is there a reason we are setting this in code as opposed to relying on the 
config?

-Grant

Reply via email to