What does it mean: It looks like second attempt to process data after the
first time it failed?
All Task AttemptsTask AttemptsMachineStatusProgressStart TimeShuffle
FinishedSort FinishedFinish TimeErrorsTask LogsCountersActions
attempt_201011081008_0002_r_000008_0/default-rack/<http://hadoop1.infolinks.local:8022>
FAILED0.00%9-Nov-2010 04:36:159-Nov-2010 05:09:08 (32mins, 52sec)9-Nov-2010
05:09:08 (0sec)9-Nov-2010 05:14:07 (37mins, 51sec)
java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
Last
4KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&start=-4097>
Last
8KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&start=-8193>
All<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&all=true>
10</taskstats.jsp?jobid=job_201011081008_0002&tipid=task_201011081008_0002_r_000008&taskid=attempt_201011081008_0002_r_000008_0>
attempt_201011081008_0002_r_000008_1/default-rack/<http://hadoop-transfer.infolinks.local:8022>
RUNNING24.98%9-Nov-2010 05:50:21
On Tue, Nov 9, 2010 at 12:58 PM, Oleg Ruchovets <[email protected]>wrote:
> Hi , running hadoop map/reduce got such exception?
>
> 1) Why does it happen?
>
> 2) Job didn't failed and continue it's execution? Does this exception cause
> losing data or map/reduce
>
> uses recovery mechanism?
>
>
>
> 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.11.87.65:50010
> 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-9208177033562590356_775948
> 2010-11-09 05:10:08,739 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to
> find target node: 10.11.87.61:50010
> 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout
> while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/10.11.87.61:55309
> remote=/10.11.87.61:50010]
> 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-10251707095594311_775950
> 2010-11-09 05:11:23,744 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to
> find target node: 10.11.87.61:50010
> 2010-11-09 05:12:29,815 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.11.87.65:50010
> 2010-11-09 05:12:29,816 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_3509928762116143133_775950
> 2010-11-09 05:12:29,818 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to
> find target node: 10.11.87.61:50010
> 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.11.87.65:50010
> 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_3138002906377068146_775950
> 2010-11-09 05:13:35,950 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to
> find target node: 10.11.87.61:50010
> 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Unable to create new block.
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2812)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>
> 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
> for block blk_3138002906377068146_775950 bad datanode[2] nodes == null
> 2010-11-09 05:13:51,758 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
> block locations. Source file
> "/user/hadoop/requests_logs/merged/2010-11-08/_temporary/_attempt_201011081008_0002_r_000008_0/part-r-00008"
> - Aborting...
> 2010-11-09 05:13:51,760 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
> 2010-11-09 05:13:54,888 INFO org.apache.hadoop.mapred.TaskRunner: Runnning
> cleanup for the task
>
>
> Thanks
>
> Oleg.
>
> ------------------------------
>
>
