It should be in the task logs for the maps. You're not seeing
"bufstart", "bufvoid", etc.? The logging is at INFO level, so if
you've set your loglevel higher that, you won't see these messages.
Are you seeing any logging from MapTask? How large are your serialized
values? -C
On Sep 10, 2008, at 1:50 PM, Florian Leibert wrote:
Hi Chris,
where do you find those values? I don't seem to see them in the
userlogs nor in the tasktracker logs...
Thanks!
Florian
On Sep 3, 2008, at 2:50 PM, Chris Douglas wrote:
InMemValBytes::reset can perform an allocation, but it should be
only as large as the value. When you look at the log for the failed
task, what does it report as the values of bufstart, bufend,
kvstart, kvend, etc. before the spill? -C
On Sep 3, 2008, at 9:49 AM, Paco NATHAN wrote:
Also, that almost always happens early in the map phase of the first
MR job which runs on our cluster.
Hadoop 0.18.1 on EC2 m1.xl instances.
We run 10 MR jobs in sequence, 6hr duration, not seeing the problem
repeated after that 1 heap space exception.
Paco
On Wed, Sep 3, 2008 at 11:42 AM, Florian Leibert <[EMAIL PROTECTED]>
wrote:
Hi,
we're running 100 XLarge instances (ec2), with a gig of heap
space for each
task - and are seeing the following error frequently (but not
always):
##### BEGIN PASTE #####
[exec] 08/09/03 11:21:09 INFO mapred.JobClient: map 43% reduce 5%
[exec] 08/09/03 11:21:16 INFO mapred.JobClient: Task Id :
attempt_200809031101_0001_m_000220_0, Status : FAILED
[exec] java.io.IOException: Spill failed
[exec] at
org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.flush(MapTask.java:688)
[exec] at org.apache.hadoop.mapred.MapTask.run(MapTask.java:
228)
[exec] at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
2209)
[exec] Caused by: java.lang.OutOfMemoryError: Java heap space
[exec] at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
$InMemValBytes.reset(MapTask.java:928)
[exec] at
org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.getVBytesForOffset(MapTask.java:891)
[exec] at
org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.sortAndSpill(MapTask.java:765)
[exec] at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access
$1600(MapTask.java:286)
[exec] at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
$SpillThread.run(MapTask.java:712)
##### END #####
Has anyone seen this? Thanks,
Florian Leibert
Sr. Software Engineer
Adknowledge Inc.
