On Jun 11, 2006, at 8:07 AM, Dennis Kubes wrote:
Can someone lead me in the right direction as to configuring settings for large sorting operations > 1M rows. I keep getting out of memory exceptions during the sort phase. Here are my current settings. I have 2G heap space on each box.
Please check the value of "mapred.child.java.opts". It controls the options for heap sized allocated to the Task. It defaults to 200m, which seems really low given that the default io.sort.mb is 100m (and yours is set to 200m). Try increasing the max heap size to 1024m or so.
-- Owen
