1) It doesn't look like I'm out of memory but it is coming really close.
2) overcommit_memory is set to 2, overcommit_ratio = 100
As for the JVM, I am using Java 1.6.
**Note of Interest**: The virtual memory I see allocated in top for each
task is more than what I am specifying in the hadoop job/site configs.
Currently each physical box has 16 GB of memory. I see the datanode and
tasktracker using:
RES VIRT
Datanode 145m 1408m
Tasktracker 206m 1439m
When idle.
So taking that into account I do 16000 MB - (1408+1439) MB which would
leave me with 13200 MB. In my old settings I was using 8 map tasks so
13200 / 8 = 1650 MB.
My mapred.child.java.opts is -Xmx1536m which should leave me a little
head room.
When running though I see some tasks reporting 1900m.
-Xavier
-----Original Message-----
From: Brian Bockelman [mailto:[EMAIL PROTECTED]
Sent: Tuesday, November 18, 2008 2:42 PM
To: [email protected]
Subject: Re: Cannot run program "bash": java.io.IOException: error=12,
Cannot allocate memory
Hey Xavier,
1) Are you out of memory (dumb question, but doesn't hurt to ask...)?
What does Ganglia tell you about the node?
2) Do you have /proc/sys/vm/overcommit_memory set to 2?
Telling Linux not to overcommit memory on Java 1.5 JVMs can be very
problematic. Java 1.5 asks for min heap size + 1 GB of reserved, non-
swap memory on Linux systems by default. The 1GB of reserved, non- swap
memory is used for the JIT to compile code; this bug wasn't fixed until
later Java 1.5 updates.
Brian
On Nov 18, 2008, at 4:32 PM, Xavier Stevens wrote:
> I'm still seeing this problem on a cluster using Hadoop 0.18.2. I
> tried
> dropping the max number of map tasks per node from 8 to 7. I still
> get
> the error although it's less frequent. But I don't get the error at
> all
> when using Hadoop 0.17.2.
>
> Anyone have any suggestions?
>
>
> -Xavier
>
> -----Original Message-----
> From: [EMAIL PROTECTED] On Behalf Of Edward J. Yoon
> Sent: Thursday, October 09, 2008 2:07 AM
> To: [email protected]
> Subject: Re: Cannot run program "bash": java.io.IOException: error=12,
> Cannot allocate memory
>
> Thanks Alexander!!
>
> On Thu, Oct 9, 2008 at 4:49 PM, Alexander Aristov
> <[EMAIL PROTECTED]> wrote:
>> I received such errors when I overloaded data nodes. You may increase
>> swap space or run less tasks.
>>
>> Alexander
>>
>> 2008/10/9 Edward J. Yoon <[EMAIL PROTECTED]>
>>
>>> Hi,
>>>
>>> I received below message. Can anyone explain this?
>>>
>>> 08/10/09 11:53:33 INFO mapred.JobClient: Task Id :
>>> task_200810081842_0004_m_000000_0, Status : FAILED
>>> java.io.IOException: Cannot run program "bash": java.io.IOException:
>>> error=12, Cannot allocate memory
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
>>> at org.apache.hadoop.util.Shell.run(Shell.java:134)
>>> at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
>>> at
>>>
> org.apache.hadoop.fs.LocalDirAllocator
> $AllocatorPerContext.getLocalPathF
> orWrite(LocalDirAllocator.java:296)
>>> at
>>>
> org
> .apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllo
> cator.java:124)
>>> at
>>>
> org
> .apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFil
> e.java:107)
>>> at
>>>
> org.apache.hadoop.mapred.MapTask
> $MapOutputBuffer.sortAndSpill(MapTask.ja
> va:734)
>>> at
>>>
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:
> 694)
>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
>>> at
>>> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
>>> 2124
>>> ) Caused by: java.io.IOException: java.io.IOException: error=12,
>>> Cannot allocate memory
>>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>>> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
>>> ... 10 more
>>>
>>> --
>>> Best regards, Edward J. Yoon
>>> [EMAIL PROTECTED]
>>> http://blog.udanax.org
>>>
>>
>>
>>
>> --
>> Best Regards
>> Alexander Aristov
>>
>
>
>
> --
> Best regards, Edward J. Yoon
> [EMAIL PROTECTED]
> http://blog.udanax.org
>
