Re: Bad connect ack with firstBadLink
Hi, Increasing the open file limit solved the issue. Thank you. On Fri, May 4, 2012 at 9:39 PM, Mapred Learn wrote: > Check your number of blocks in the cluster. > > This also indicates that your datanodes are more full than they should be. > > Try deleting unnecessary blocks. > > On Fri, May 4, 2012 at 7:40 AM, Mohit Anchlia >wrote: > > > Please see: > > > > http://hbase.apache.org/book.html#dfs.datanode.max.xcievers > > > > On Fri, May 4, 2012 at 5:46 AM, madhu phatak > wrote: > > > > > Hi, > > > We are running a three node cluster . From two days whenever we copy > file > > > to hdfs , it is throwing java.IO.Exception Bad connect ack with > > > firstBadLink . I searched in net, but not able to resolve the issue. > The > > > following is the stack trace from datanode log > > > > > > 2012-05-04 18:08:08,868 INFO > > > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > > > blk_-7520371350112346377_50118 received exception > > java.net.SocketException: > > > Connection reset > > > 2012-05-04 18:08:08,869 ERROR > > > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > > > 172.23.208.17:50010, > > > storageID=DS-1340171424-172.23.208.17-50010-1334672673051, > > infoPort=50075, > > > ipcPort=50020):DataXceiver > > > java.net.SocketException: Connection reset > > >at java.net.SocketInputStream.read(SocketInputStream.java:168) > > >at > java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > > >at > java.io.BufferedInputStream.read(BufferedInputStream.java:317) > > >at java.io.DataInputStream.read(DataInputStream.java:132) > > >at > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:262) > > >at > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:309) > > >at > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:373) > > >at > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:525) > > >at > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357) > > >at > > > > > > > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) > > >at java.lang.Thread.run(Thread.java:662) > > > > > > > > > It will be great if some one can point to the direction how to solve > this > > > problem. > > > > > > -- > > > https://github.com/zinnia-phatak-dev/Nectar > > > > > > -- https://github.com/zinnia-phatak-dev/Nectar
Re: Bad connect ack with firstBadLink
Check your number of blocks in the cluster. This also indicates that your datanodes are more full than they should be. Try deleting unnecessary blocks. On Fri, May 4, 2012 at 7:40 AM, Mohit Anchlia wrote: > Please see: > > http://hbase.apache.org/book.html#dfs.datanode.max.xcievers > > On Fri, May 4, 2012 at 5:46 AM, madhu phatak wrote: > > > Hi, > > We are running a three node cluster . From two days whenever we copy file > > to hdfs , it is throwing java.IO.Exception Bad connect ack with > > firstBadLink . I searched in net, but not able to resolve the issue. The > > following is the stack trace from datanode log > > > > 2012-05-04 18:08:08,868 INFO > > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > > blk_-7520371350112346377_50118 received exception > java.net.SocketException: > > Connection reset > > 2012-05-04 18:08:08,869 ERROR > > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > > 172.23.208.17:50010, > > storageID=DS-1340171424-172.23.208.17-50010-1334672673051, > infoPort=50075, > > ipcPort=50020):DataXceiver > > java.net.SocketException: Connection reset > >at java.net.SocketInputStream.read(SocketInputStream.java:168) > >at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) > >at java.io.BufferedInputStream.read(BufferedInputStream.java:317) > >at java.io.DataInputStream.read(DataInputStream.java:132) > >at > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:262) > >at > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:309) > >at > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:373) > >at > > > > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:525) > >at > > > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357) > >at > > > > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) > >at java.lang.Thread.run(Thread.java:662) > > > > > > It will be great if some one can point to the direction how to solve this > > problem. > > > > -- > > https://github.com/zinnia-phatak-dev/Nectar > > >
Re: Bad connect ack with firstBadLink
Please see: http://hbase.apache.org/book.html#dfs.datanode.max.xcievers On Fri, May 4, 2012 at 5:46 AM, madhu phatak wrote: > Hi, > We are running a three node cluster . From two days whenever we copy file > to hdfs , it is throwing java.IO.Exception Bad connect ack with > firstBadLink . I searched in net, but not able to resolve the issue. The > following is the stack trace from datanode log > > 2012-05-04 18:08:08,868 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock > blk_-7520371350112346377_50118 received exception java.net.SocketException: > Connection reset > 2012-05-04 18:08:08,869 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 172.23.208.17:50010, > storageID=DS-1340171424-172.23.208.17-50010-1334672673051, infoPort=50075, > ipcPort=50020):DataXceiver > java.net.SocketException: Connection reset >at java.net.SocketInputStream.read(SocketInputStream.java:168) >at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) >at java.io.BufferedInputStream.read(BufferedInputStream.java:317) >at java.io.DataInputStream.read(DataInputStream.java:132) >at > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:262) >at > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:309) >at > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:373) >at > > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:525) >at > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357) >at > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) >at java.lang.Thread.run(Thread.java:662) > > > It will be great if some one can point to the direction how to solve this > problem. > > -- > https://github.com/zinnia-phatak-dev/Nectar >
Bad connect ack with firstBadLink
Hi, We are running a three node cluster . From two days whenever we copy file to hdfs , it is throwing java.IO.Exception Bad connect ack with firstBadLink . I searched in net, but not able to resolve the issue. The following is the stack trace from datanode log 2012-05-04 18:08:08,868 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_-7520371350112346377_50118 received exception java.net.SocketException: Connection reset 2012-05-04 18:08:08,869 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( 172.23.208.17:50010, storageID=DS-1340171424-172.23.208.17-50010-1334672673051, infoPort=50075, ipcPort=50020):DataXceiver java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:262) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:309) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:373) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:525) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:357) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103) at java.lang.Thread.run(Thread.java:662) It will be great if some one can point to the direction how to solve this problem. -- https://github.com/zinnia-phatak-dev/Nectar
Re: java.io.IOException: Bad connect ack with firstBadLink
What does it mean: It looks like second attempt to process data after the first time it failed? All Task AttemptsTask AttemptsMachineStatusProgressStart TimeShuffle FinishedSort FinishedFinish TimeErrorsTask LogsCountersActions attempt_201011081008_0002_r_08_0/default-rack/<http://hadoop1.infolinks.local:8022> FAILED0.00%9-Nov-2010 04:36:159-Nov-2010 05:09:08 (32mins, 52sec)9-Nov-2010 05:09:08 (0sec)9-Nov-2010 05:14:07 (37mins, 51sec) java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262) Last 4KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_08_0&start=-4097> Last 8KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_08_0&start=-8193> All<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_08_0&all=true> 10 attempt_201011081008_0002_r_08_1/default-rack/<http://hadoop-transfer.infolinks.local:8022> RUNNING24.98%9-Nov-2010 05:50:21 On Tue, Nov 9, 2010 at 12:58 PM, Oleg Ruchovets wrote: > Hi , running hadoop map/reduce got such exception? > > 1) Why does it happen? > > 2) Job didn't failed and continue it's execution? Does this exception cause > losing data or map/reduce > > uses recovery mechanism? > > > > 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Exception in > createBlockOutputStream java.io.IOException: Bad connect ack with > firstBadLink 10.11.87.65:50010 > 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning > block blk_-9208177033562590356_775948 > 2010-11-09 05:10:08,739 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to > find target node: 10.11.87.61:50010 > 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Exception in > createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout > while waiting for channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/10.11.87.61:55309 > remote=/10.11.87.61:50010] > 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning > block blk_-10251707095594311_775950 > 2010-11-09 05:11:23,744 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to > find target node: 10.11.87.61:50010 > 2010-11-09 05:12:29,815 INFO org.apache.hadoop.hdfs.DFSClient: Exception in > createBlockOutputStream java.io.IOException: Bad connect ack with > firstBadLink 10.11.87.65:50010 > 2010-11-09 05:12:29,816 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning > block blk_3509928762116143133_775950 > 2010-11-09 05:12:29,818 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to > find target node: 10.11.87.61:50010 > 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Exception in > createBlockOutputStream java.io.IOException: Bad connect ack with > firstBadLink 10.11.87.65:50010 > 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning > block blk_3138002906377068146_775950 > 2010-11-09 05:13:35,950 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to > find target node: 10.11.87.61:50010 > 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer > Exception: java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2812) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262) > > 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery > for block blk_3138002906377068146_775950 bad datanode[2] nodes == null > 2010-11-09 05:13:51,758 WARN org.apache.hadoop.hdfs.DFSClient: Could not get > block locations. Source file > "/user/hadoop/requests_logs/merged/2010-11-08/_temporary/_attempt_201011081008_0002_r_08_0/part-r-8" > - Aborting... > 2010-11-09 05:13:51,760 WARN org.apache.hadoop.mapred.TaskTracker: Error > running child > java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010 > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262) > 2010-11-09 05:13:54,888 INFO org.apache.hadoop.mapred.TaskRunner: Runnning > cleanup for the task > > > Thanks > > Oleg. > > -- > >
java.io.IOException: Bad connect ack with firstBadLink
Hi , running hadoop map/reduce got such exception? 1) Why does it happen? 2) Job didn't failed and continue it's execution? Does this exception cause losing data or map/reduce uses recovery mechanism? 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-9208177033562590356_775948 2010-11-09 05:10:08,739 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.87.61:55309 remote=/10.11.87.61:50010] 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-10251707095594311_775950 2010-11-09 05:11:23,744 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010 2010-11-09 05:12:29,815 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010 2010-11-09 05:12:29,816 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_3509928762116143133_775950 2010-11-09 05:12:29,818 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_3138002906377068146_775950 2010-11-09 05:13:35,950 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2812) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262) 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_3138002906377068146_775950 bad datanode[2] nodes == null 2010-11-09 05:13:51,758 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/user/hadoop/requests_logs/merged/2010-11-08/_temporary/_attempt_201011081008_0002_r_08_0/part-r-8" - Aborting... 2010-11-09 05:13:51,760 WARN org.apache.hadoop.mapred.TaskTracker: Error running child java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010 at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262) 2010-11-09 05:13:54,888 INFO org.apache.hadoop.mapred.TaskRunner: Runnning cleanup for the task Thanks Oleg. --
RE: hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink
I have resolved the issue: What i did: 1) '/etc/init.d/iptables stop' -->stopped firewall 2) SELINUX=disabled in '/etc/selinux/config' file.-->disabled selinux I worked for me after these two changes. thanks, --umer > From: m_umer_ars...@hotmail.com > To: common-user@hadoop.apache.org > Subject: hdfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > Date: Mon, 31 Aug 2009 23:35:31 + > > > Hi, > > I have set-up an 8-node private hadoop cluster having IP-addresses as follows: > > 192.168.1.10 (master) > 192.168.1.11 > 192.168.1.12 > 192.168.1.13 > 192.168.1.14 > 192.168.1.15 > 192.168.1.16 > 192.168.1.17 > > Address 192.168.1.10, master, is running NN+JT and all other nodes are slaves > i.e. running DN+TT. I am trying to put data on HDFS using command: hadoop dfs > -put 8GB_input 8GB_input > > I have noticed that some blocks are not replicated/placed on nodes with IP > addresses 192.168.1.11, 192.168.1.15, and 192.168.1.16 and i get the > following error messages: > > $ hadoop dfs -put 8GB_input 8GB_input > 09/08/31 18:25:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 192.168.1.11:50010 > 09/08/31 18:25:45 INFO hdfs.DFSClient: Abandoning block > blk_-8575812198227241296_1001 > 09/08/31 18:25:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 192.168.1.16:50010 > 09/08/31 18:25:51 INFO hdfs.DFSClient: Abandoning block > blk_-2932256218448902464_1001 > 09/08/31 18:25:57 INFO hdfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 192.168.1.11:50010 > 09/08/31 18:25:57 INFO hdfs.DFSClient: Abandoning block > blk_-1014449966480421244_1001 > 09/08/31 18:26:03 INFO hdfs.DFSClient: Exception in createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 192.168.1.16:50010 > 09/08/31 18:26:03 INFO hdfs.DFSClient: Abandoning block > blk_7193173823538206978_1001 > 09/08/31 18:26:09 WARN hdfs.DFSClient: DataStreamer Exception: > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2731) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2182) > > 09/08/31 18:26:09 WARN hdfs.DFSClient: Error Recovery for block > blk_7193173823538206978_1001 bad datanode[2] nodes == null > 09/08/31 18:26:09 WARN hdfs.DFSClient: Could not get block locations. Source > file "/user/umer/8GB_input" - Aborting... > put: Bad connect ack with firstBadLink 192.168.1.16:50010 > - > > Sometimes the input file is replicated successfully (excluding these three > nodes) and sometimes the copy process i.e. 'hdfs -put input input' terminates. > > NOTE: My replication factor = 3. > > I am able to see that all machines are up-and-running using Web-UI > http://192.168.1.10:50070. > > I will be greatful for any suggestion/comment in this regard. > > thanks, > --umer > > > > > _ > Share your memories online with anyone you want. > http://www.microsoft.com/middleeast/windows/windowslive/products/photos-share.aspx?tab=1 _ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/
hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink
Hi, I have set-up an 8-node private hadoop cluster having IP-addresses as follows: 192.168.1.10 (master) 192.168.1.11 192.168.1.12 192.168.1.13 192.168.1.14 192.168.1.15 192.168.1.16 192.168.1.17 Address 192.168.1.10, master, is running NN+JT and all other nodes are slaves i.e. running DN+TT. I am trying to put data on HDFS using command: hadoop dfs -put 8GB_input 8GB_input I have noticed that some blocks are not replicated/placed on nodes with IP addresses 192.168.1.11, 192.168.1.15, and 192.168.1.16 and i get the following error messages: $ hadoop dfs -put 8GB_input 8GB_input 09/08/31 18:25:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.1.11:50010 09/08/31 18:25:45 INFO hdfs.DFSClient: Abandoning block blk_-8575812198227241296_1001 09/08/31 18:25:51 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.1.16:50010 09/08/31 18:25:51 INFO hdfs.DFSClient: Abandoning block blk_-2932256218448902464_1001 09/08/31 18:25:57 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.1.11:50010 09/08/31 18:25:57 INFO hdfs.DFSClient: Abandoning block blk_-1014449966480421244_1001 09/08/31 18:26:03 INFO hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 192.168.1.16:50010 09/08/31 18:26:03 INFO hdfs.DFSClient: Abandoning block blk_7193173823538206978_1001 09/08/31 18:26:09 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2731) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2182) 09/08/31 18:26:09 WARN hdfs.DFSClient: Error Recovery for block blk_7193173823538206978_1001 bad datanode[2] nodes == null 09/08/31 18:26:09 WARN hdfs.DFSClient: Could not get block locations. Source file "/user/umer/8GB_input" - Aborting... put: Bad connect ack with firstBadLink 192.168.1.16:50010 - Sometimes the input file is replicated successfully (excluding these three nodes) and sometimes the copy process i.e. 'hdfs -put input input' terminates. NOTE: My replication factor = 3. I am able to see that all machines are up-and-running using Web-UI http://192.168.1.10:50070. I will be greatful for any suggestion/comment in this regard. thanks, --umer _ Share your memories online with anyone you want. http://www.microsoft.com/middleeast/windows/windowslive/products/photos-share.aspx?tab=1
