It is in a map phase. I don't think I used a custom chunker. My splits are set to be 128M.
On Fri, May 23, 2008 at 9:07 AM, pi song <[EMAIL PROTECTED]> wrote: > Dear Tanton, > > This means the MemoryManager is not successful at reclaiming memory. Did > that happen in Map phase or Reduce phase? Did you use a custom chunker? How > big is your split? > > Pi > > On Fri, May 23, 2008 at 3:52 PM, Tanton Gibbs <[EMAIL PROTECTED]> > wrote: > >> I upgraded to hadoop 17 and the latest Pig from svn. >> >> I'm now getting a ton of lines in my log files that say: >> >> 2008-05-23 00:49:27,832 INFO >> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler >> called init = 1441792(1408K) used = 483176072(471851K) committed = >> 641335296(626304K) max = 954466304(932096K) >> >> In addition, jobs on big files are running very slowly. >> >> Does anyone have any ideas as to what I could have screwed up? >> >> Thanks! >> Tanton >> >
