HI David. an you explain in a bit more detail what was the issue? Thanks. Shahab
On Tue, May 14, 2013 at 2:29 AM, David Parks <[email protected]> wrote: > I just hate it when I figure out a problem right after asking for help.*** > * > > ** ** > > Finding the task logs via the task tracker website identified the problem > which didn’t show up elsewhere. Simple mis-configuration which I did > concurrently with the DNS update that threw me off track.**** > > ** ** > > Dave**** > > ** ** > > ** ** > > *From:* David Parks [mailto:[email protected]] > *Sent:* Tuesday, May 14, 2013 1:20 PM > *To:* [email protected] > *Subject:* JobClient: Error reading task output - after instituting a DNS > server**** > > ** ** > > So we just configured a local DNS server for hostname resolution and > stopped using a hosts file and now jobs fail on us. But I can’t figure out > why.**** > > ** ** > > You can see the error below, but if I run curl to any of those URLs they > come back “Failed to retrieve stdout log”, which doesn’t look much like a > DNS issue.**** > > ** ** > > I can ping and do nslookup from any host to any other host. This is a CDH4 > cluster and the host inspector is happy as could be; also Cloudera Manager > indicates all is well.**** > > ** ** > > When I open the task tracker website I see the first task attempt show up > on the site there for maybe 10 seconds or so before it fails.**** > > ** ** > > Any idea what I need to look at here?**** > > ** ** > > Job:**** > > ====**** > > 13/05/14 05:13:40 INFO input.FileInputFormat: Total input paths to process > : 131**** > > 13/05/14 05:13:41 INFO input.FileInputFormat: Total input paths to process > : 1**** > > 13/05/14 05:13:42 INFO mapred.JobClient: Running job: job_201305131758_0003 > **** > > 13/05/14 05:13:43 INFO mapred.JobClient: map 0% reduce 0%**** > > 13/05/14 05:13:47 INFO mapred.JobClient: Task Id : > attempt_201305131758_0003_m_000353_0, Status : FAILED**** > > java.lang.Throwable: Child Error**** > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)*** > * > > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > **** > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)*** > * > > ** ** > > 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task > outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout > **** > > 13/05/14 05:13:47 WARN mapred.JobClient: Error reading task > outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stderr > **** > > 13/05/14 05:13:50 INFO mapred.JobClient: Task Id : > attempt_201305131758_0003_r_000521_0, Status : FAILED**** > > java.lang.Throwable: Child Error**** > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)*** > * > > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > **** > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)*** > * > > ** ** > > 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task > outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stdout > **** > > 13/05/14 05:13:50 WARN mapred.JobClient: Error reading task > outputhttp://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_r_000521_0&filter=stderr > **** > > ** ** > > ** ** > > curl of above URL:**** > > ====================**** > > davidparks21@hadoop-meta1:~$ curl ' > http://hadoop-fullslot2:50060/tasklog?plaintext=true&attemptid=attempt_201305131758_0003_m_000353_0&filter=stdout > '**** > > <html>**** > > <head>**** > > <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>* > *** > > <title>Error 410 Failed to retrieve stdout log for task: > attempt_201305131758_0003_m_000353_0</title>**** > > </head>**** > > <body><h2>HTTP ERROR 410</h2>**** > > <p>Problem accessing /tasklog. Reason:**** > > <pre> Failed to retrieve stdout log for task: > attempt_201305131758_0003_m_000353_0</pre></p><hr /><i><small>Powered by > Jetty://</small></i><br/> **** > > <br/>**** > > <br/>**** > > <br/>**** > > <br/>**** >
