Sorry to bug you guys again but I found the problem. An old hadoop-site that was in the class path and had limit on the "mapred.child.ulimit" to 500000
Thanks -Yair -----Original Message----- From: Yair Even-Zohar [mailto:[email protected]] Sent: Tuesday, August 11, 2009 4:11 PM To: [email protected]; [email protected] Subject: problem setting mapred.child.java.opts I'm running a mapreduce using Hbase table as input with some distributed cache file and all works well. However, when I set: c.set("mapred.child.java.opts", "-Xmx512m") in the java code and using the exact same input and exact same distributed cache I'm getting the following: on the master side: 09/08/11 08:19:05 INFO mapred.JobClient: Task Id : attempt_200908110722_0016_m_000001_0, Status : FAILED java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418) 09/08/11 08:19:05 WARN mapred.JobClient: Error reading task outputhttp://domU-12-31-39-02-95-D2.compute-1.internal:50060/tasklog?pla intext=true&taskid=attempt_200908110722_0016_m_000001_0&filter=stdout 09/08/11 08:19:05 WARN mapred.JobClient: Error reading task outputhttp://domU-12-31-39-02-95-D2.compute-1.internal:50060/tasklog?pla intext=true&taskid=attempt_200908110722_0016_m_000001_0&filter=stderr On the slave log I'm getting as if I do not have enough space. But I run these on EC2 with 4 GB or memory so that's insane. Error occurred during initialization of VM Could not reserve enough space for object heap # # An unexpected error has been detected by Java Runtime Environment: # # SIGSEGV (0xb) at pc=0xf7ec40da, pid=4948, tid=4158831504 # # Java VM: Java HotSpot(TM) Server VM (10.0-b23 mixed mode linux-x86) # Problematic frame: # C [libc.so.6+0x6d0da] cfree+0x7a # # An error report file with more information is saved as: # /mnt/hadoop/mapred/local/taskTracker/jobcache/job_200908110722_0016/atte mpt_200908110722_0016_m_000001_0/work/hs_err_pid4948.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # Furthermore, I have checked the job_XXXXXXXXX_YYYY_conf.xml and the only difference between the two runs (successful and non-successful ) is the size < <property><name>mapred.child.java.opts</name><value>-Xmx512m</value></pr operty> > <property><name>mapred.child.java.opts</name><value>-Xmx200m</value></pr operty> Any idea regarding this proble is welcome Thanks -Yair
