Hi Rohini, The similar problem was just encountered for me yesterday. But for my situation, the max process num (ulimit -u) is set to 1024, which is too small. And when i increase it to 1000000, the problem gone. But u said "Ulimit on the machine is set to unlimited", i'm not sure this will help or not :)
And also check about `cat /proc/sys/kernel/threads-max', this seems to be a system-wide setting for total number of threads. On Tue, Mar 6, 2012 at 4:30 AM, Rohini U <rohin...@gmail.com> wrote: > Hi All, > > I am running a map reduce job that uses around 120 MB of data and I get > this out of memory error. Ulimit on the machine is set to unlimited. Any > ideas on how to fix this? > The stack trace is as given below: > > > Exception in thread "main" org.apache.hadoop.ipc.RemoteException: > java.io.IOException: java.lang.OutOfMemoryError: unable to create new > native thread > at java.lang.Thread.start0(Native Method) > at java.lang.Thread.start(Thread.java:597) > at > org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.kill(JvmManager.java:553) > at > org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvmRunner(JvmManager.java:317) > at > org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvm(JvmManager.java:297) > at > org.apache.hadoop.mapred.JvmManager$JvmManagerForType.taskKilled(JvmManager.java:289) > at > org.apache.hadoop.mapred.JvmManager.taskKilled(JvmManager.java:158) > at org.apache.hadoop.mapred.TaskRunner.kill(TaskRunner.java:782) > at > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:2938) > at > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:2910) > at > org.apache.hadoop.mapred.TaskTracker.purgeTask(TaskTracker.java:1974) > at > org.apache.hadoop.mapred.TaskTracker.fatalError(TaskTracker.java:3327) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) > > at org.apache.hadoop.ipc.Client.call(Client.java:1107) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) > at $Proxy0.fatalError(Unknown Source) > at org.apache.hadoop.mapred.Child.main(Child.java:325) > > > > Thanks > -Rohini > -- Kindest Regards, Clay Chiang
