Hey Xavier,
1) Are you out of memory (dumb question, but doesn't hurt to ask...)?
What does Ganglia tell you about the node?
2) Do you have /proc/sys/vm/overcommit_memory set to 2?
Telling Linux not to overcommit memory on Java 1.5 JVMs can be very
problematic. Java 1.5 asks for min heap size + 1 GB of reserved, non-
swap memory on Linux systems by default. The 1GB of reserved, non-
swap memory is used for the JIT to compile code; this bug wasn't fixed
until later Java 1.5 updates.
Brian
On Nov 18, 2008, at 4:32 PM, Xavier Stevens wrote:
I'm still seeing this problem on a cluster using Hadoop 0.18.2. I
tried
dropping the max number of map tasks per node from 8 to 7. I still
get
the error although it's less frequent. But I don't get the error at
all
when using Hadoop 0.17.2.
Anyone have any suggestions?
-Xavier
-----Original Message-----
From: [EMAIL PROTECTED] On Behalf Of Edward J. Yoon
Sent: Thursday, October 09, 2008 2:07 AM
To: [email protected]
Subject: Re: Cannot run program "bash": java.io.IOException: error=12,
Cannot allocate memory
Thanks Alexander!!
On Thu, Oct 9, 2008 at 4:49 PM, Alexander Aristov
<[EMAIL PROTECTED]> wrote:
I received such errors when I overloaded data nodes. You may increase
swap space or run less tasks.
Alexander
2008/10/9 Edward J. Yoon <[EMAIL PROTECTED]>
Hi,
I received below message. Can anyone explain this?
08/10/09 11:53:33 INFO mapred.JobClient: Task Id :
task_200810081842_0004_m_000000_0, Status : FAILED
java.io.IOException: Cannot run program "bash": java.io.IOException:
error=12, Cannot allocate memory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
at
org.apache.hadoop.fs.LocalDirAllocator
$AllocatorPerContext.getLocalPathF
orWrite(LocalDirAllocator.java:296)
at
org
.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllo
cator.java:124)
at
org
.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFil
e.java:107)
at
org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.sortAndSpill(MapTask.ja
va:734)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:
694)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:
2124
) Caused by: java.io.IOException: java.io.IOException: error=12,
Cannot allocate memory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 10 more
--
Best regards, Edward J. Yoon
[EMAIL PROTECTED]
http://blog.udanax.org
--
Best Regards
Alexander Aristov
--
Best regards, Edward J. Yoon
[EMAIL PROTECTED]
http://blog.udanax.org
