I've seen those errors when I was playing the values in the core-site.xml, dfs-site.xml and mapreduce-site.xml.
Might be worth comparing your values to mine discussed in the thread http://www.mail-archive.com/[email protected]/msg02522.html which also represent 8G DN machines Cheers Tim On Wed, Oct 28, 2009 at 4:41 PM, Hassaan Khan <[email protected]> wrote: > I'm running Hadoop 0.20.1+133 (Cloudera distro) > I tried setting up a multi-node Hadoop cluster and on executing the command: > hadoop jar /usr/lib/hadoop/hadoop-0.20.1+133-examples.jar grep input output > 'dfs[a-z.]+' > I get: > > 09/10/27 20:39:21 INFO mapred.FileInputFormat: Total input paths to process > : 5 > 09/10/27 20:39:21 INFO mapred.JobClient: Running job: job_200910272023_0002 > 09/10/27 20:39:22 INFO mapred.JobClient: map 0% reduce 0% > 09/10/27 20:39:30 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_m_000006_0, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:39:30 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_0&filter=stdout > 09/10/27 20:39:30 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_0&filter=stderr > 09/10/27 20:39:36 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_r_000020_0, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:39:36 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_0&filter=stdout > 09/10/27 20:39:36 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_0&filter=stderr > 09/10/27 20:39:42 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_m_000006_1, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:39:42 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_1&filter=stdout > 09/10/27 20:39:42 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_1&filter=stderr > 09/10/27 20:39:48 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_r_000020_1, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:39:48 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_1&filter=stdout > 09/10/27 20:39:48 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_1&filter=stderr > 09/10/27 20:39:57 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_m_000006_2, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:39:57 WARN mapred.JobClient: Error reading task outputhttp:// > anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_2&filter=stdout > 09/10/27 20:39:57 WARN mapred.JobClient: Error reading task outputhttp:// > anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000006_2&filter=stderr > 09/10/27 20:40:03 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_r_000020_2, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:40:03 WARN mapred.JobClient: Error reading task outputhttp:// > anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_2&filter=stdout > 09/10/27 20:40:03 WARN mapred.JobClient: Error reading task outputhttp:// > anza4.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000020_2&filter=stderr > 09/10/27 20:40:15 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_m_000005_0, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:40:15 WARN mapred.JobClient: Error reading task outputhttp:// > anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_0&filter=stdout > 09/10/27 20:40:15 WARN mapred.JobClient: Error reading task outputhttp:// > anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_0&filter=stderr > 09/10/27 20:40:21 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_r_000019_0, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:40:21 WARN mapred.JobClient: Error reading task outputhttp:// > anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_0&filter=stdout > 09/10/27 20:40:21 WARN mapred.JobClient: Error reading task outputhttp:// > anza2.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_0&filter=stderr > 09/10/27 20:40:30 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_m_000005_1, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:40:30 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_1&filter=stdout > 09/10/27 20:40:30 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_1&filter=stderr > 09/10/27 20:40:36 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_r_000019_1, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:40:36 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_1&filter=stdout > 09/10/27 20:40:36 WARN mapred.JobClient: Error reading task outputhttp:// > anza5.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_1&filter=stderr > 09/10/27 20:40:42 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_m_000005_2, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:40:42 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_2&filter=stdout > 09/10/27 20:40:42 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_m_000005_2&filter=stderr > 09/10/27 20:40:48 INFO mapred.JobClient: Task Id : > attempt_200910272023_0002_r_000019_2, Status : FAILED > java.lang.Throwable: Child Error > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:471) > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:458) > > 09/10/27 20:40:48 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_2&filter=stdout > 09/10/27 20:40:48 WARN mapred.JobClient: Error reading task outputhttp:// > anza3.eng.blah.com:50060/tasklog?plaintext=true&taskid=attempt_200910272023_0002_r_000019_2&filter=stderr > 09/10/27 20:40:57 INFO mapred.JobClient: Job complete: job_200910272023_0002 > 09/10/27 20:40:57 INFO mapred.JobClient: Counters: 0 > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1293) > at org.apache.hadoop.examples.Grep.run(Grep.java:69) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.examples.Grep.main(Grep.java:93) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) > at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:185) > > > > Based upon a post I read to a similar issue, I changed my /etc/hosts file > to: > > # Do not remove the following line, or various programs > # that require network functionality will fail. > 127.0.0.1 localhost.localdomain localhost > ::1 localhost6.localdomain6 localhost6 > 10.50.65.61 anza1.eng.blah.com anza1 > 10.50.65.62 anza2.eng.blah.com anza2 > 10.50.65.63 anza3.eng.blah.com anza3 > 10.50.65.64 anza4.eng.blah.com anza4 > 10.50.65.65 anza5.eng.blah.com anza5 > > > > Also, when I look at: > /var/log/hadoop/userlogs/attempt_200910271659_0007_r_000019_0 on a slave > STDOUT: > Error occurred during initialization of VM > Could not reserve enough space for object heap > STDERR: > Could not create the Java virtual machine. > > > My slaves are running on boxes with 8GB or RAM and under: > JAVA_HEAP_MAX=-Xmx1000m > > And under mapred-site.xml: > <property> > <name>mapred.child.java.opts</name> > <value>-Xmx2048m</value> > </property> > > > I can't figure out why the slaves are failing? >
