I just hate it when I figure out a problem right after asking for help.
Finding the task logs via the task tracker website identified the problem which didn't show up elsewhere. Simple mis-configuration which I did concurrently with the DNS update that threw me off track. Dave From: David Parks [mailto:[email protected]] Sent: Tuesday, May 14, 2013 1:20 PM To: [email protected] Subject: JobClient: Error reading task output - after instituting a DNS server So we just configured a local DNS server for hostname resolution and stopped using a hosts file and now jobs fail on us. But I can't figure out why. You can see the error below, but if I run curl to any of those URLs they come back "Failed to retrieve stdout log", which doesn't look much like a DNS issue. I can ping and do nslookup from any host to any other host. This is a CDH4 cluster and the host inspector is happy as could be; also Cloudera Manager indicates all is well. When I open the task tracker website I see the first task attempt show up on the site there for maybe 10 seconds or so before it fails. Any idea what I need to look at here? Job: ==== 13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process : 131 13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process : 1 13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003 13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0% 13/05/14 05:13:47 INFO mapred.JobClient: Task Id : attempt_201305131758_0003_m_000353_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237) 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_m_000353_0&filter=stdout 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_m_000353_0&filter=stderr 13/05/14 05:13:50 INFO mapred.JobClient: Task Id : attempt_201305131758_0003_r_000521_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250) Caused by: java.io.IOException: Task process exit with nonzero status of 1. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237) 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_r_000521_0&filter=stdout 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt _201305131758_0003_r_000521_0&filter=stderr curl of above URL: ==================== davidparks21@hadoop-meta1:~$ curl 'http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_2013 05131758_0003_m_000353_0&filter=stdout' <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/> <title>Error 410 Failed to retrieve stdout log for task: attempt_201305131758_0003_m_000353_0</title> </head> <body><h2>HTTP ERROR 410</h2> <p>Problem accessing /tasklog. Reason: <pre> Failed to retrieve stdout log for task: attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> <br/> <br/> <br/> <br/>
