I upped my maximum memory from 1024M to 2048M and the problem went away. I think the problem was that my sortable memory was already set to 400M so it was very close to the 50% mark already.
Is there a way to up the spillable threshold to 80%? On Fri, May 23, 2008 at 10:04 AM, Tanton Gibbs <[EMAIL PROTECTED]> wrote: > It is in a map phase. I don't think I used a custom chunker. My > splits are set to be 128M. > > On Fri, May 23, 2008 at 9:07 AM, pi song <[EMAIL PROTECTED]> wrote: >> Dear Tanton, >> >> This means the MemoryManager is not successful at reclaiming memory. Did >> that happen in Map phase or Reduce phase? Did you use a custom chunker? How >> big is your split? >> >> Pi >> >> On Fri, May 23, 2008 at 3:52 PM, Tanton Gibbs <[EMAIL PROTECTED]> >> wrote: >> >>> I upgraded to hadoop 17 and the latest Pig from svn. >>> >>> I'm now getting a ton of lines in my log files that say: >>> >>> 2008-05-23 00:49:27,832 INFO >>> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler >>> called init = 1441792(1408K) used = 483176072(471851K) committed = >>> 641335296(626304K) max = 954466304(932096K) >>> >>> In addition, jobs on big files are running very slowly. >>> >>> Does anyone have any ideas as to what I could have screwed up? >>> >>> Thanks! >>> Tanton >>> >> >
