Hi Tanton,
I am having the same problem, but I have got an Out of Memory Exception
in the reduce phase. Which Hadoop config parameter did you change? Is it
the io.seqfile.compress.blocksize?
My current value for this parameter is:
<property>
<name>io.seqfile.compress.blocksize</name>
<value>1000000</value>
<description>The minimum block size for compression in block compressed
SequenceFiles.
</description>
</property>
The logs I got:
2008-05-27 11:05:38,087 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 273852568(267434K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:38,802 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 273860104(267441K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:44,893 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 279642088(273087K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:52,704 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 217972656(212863K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:05:56,510 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 269271376(262960K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:03,686 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 244418296(238689K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:10,610 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 269740120(263418K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:16,370 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 271831992(265460K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:19,813 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 258029960(251982K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:23,948 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 279642104(273087K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:27,208 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 195321216(190743K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:30,932 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 266489648(260243K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:34,463 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 279642088(273087K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:38,214 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 279642112(273088K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:06:44,571 INFO org.apache.pig.impl.util.SpillableMemoryManager:
low memory handler called init = 5439488(5312K) used = 268382184(262091K)
committed = 279642112(273088K) max = 279642112(273088K)
2008-05-27 11:11:02,570 INFO org.apache.hadoop.mapred.TaskRunner: Communication
exception: java.lang.OutOfMemoryError: Java heap space
2008-05-27 11:11:02,571 ERROR
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce:
java.lang.OutOfMemoryError: Java heap space
2008-05-27 11:11:03,234 INFO org.apache.hadoop.ipc.Client:
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at org.apache.hadoop.ipc.Client$Connection$1.read(Client.java:190)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.io.DataInputStream.readInt(DataInputStream.java:370)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:276)
Thanks,
Iván
El vie, 23-05-2008 a las 10:23 -0500, Tanton Gibbs escribió:
> I upped my maximum memory from 1024M to 2048M and the problem went
> away. I think the problem was that my sortable memory was already set
> to 400M so it was very close to the 50% mark already.
>
> Is there a way to up the spillable threshold to 80%?
>
> On Fri, May 23, 2008 at 10:04 AM, Tanton Gibbs <[EMAIL PROTECTED]> wrote:
> > It is in a map phase. I don't think I used a custom chunker. My
> > splits are set to be 128M.
> >
> > On Fri, May 23, 2008 at 9:07 AM, pi song <[EMAIL PROTECTED]> wrote:
> >> Dear Tanton,
> >>
> >> This means the MemoryManager is not successful at reclaiming memory. Did
> >> that happen in Map phase or Reduce phase? Did you use a custom chunker? How
> >> big is your split?
> >>
> >> Pi
> >>
> >> On Fri, May 23, 2008 at 3:52 PM, Tanton Gibbs <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >>> I upgraded to hadoop 17 and the latest Pig from svn.
> >>>
> >>> I'm now getting a ton of lines in my log files that say:
> >>>
> >>> 2008-05-23 00:49:27,832 INFO
> >>> org.apache.pig.impl.util.SpillableMemoryManager: low memory handler
> >>> called init = 1441792(1408K) used = 483176072(471851K) committed =
> >>> 641335296(626304K) max = 954466304(932096K)
> >>>
> >>> In addition, jobs on big files are running very slowly.
> >>>
> >>> Does anyone have any ideas as to what I could have screwed up?
> >>>
> >>> Thanks!
> >>> Tanton
> >>>
> >>
> >
