Hi Chris, Thanks a lot for the help. But after lot of investigation I found that the issue was with the cached socket connection which was raised as a bug by Nicholas. Bug details are as follows,
HDFS-3373 <https://issues.apache.org/jira/browse/HDFS-3373> FileContext HDFS implementation can leak socket caches When I executed command netstat -a |grep 50010 the count was approximately 52000. This issue is fixed in 0.20.3<https://issues.apache.org/jira/browse/HDFS/fixforversion/12314814>, 0.20.205.0<https://issues.apache.org/jira/browse/HDFS/fixforversion/12316392>, but its not present in hadoop-1.2.X. Could you please guide me as to what could I do.? -Divya On Sat, Oct 26, 2013 at 12:38 AM, Chris Nauroth <cnaur...@hortonworks.com>wrote: > Hi Divya, > > The exceptions indicate that the HDFS client failed to establish a network > connection to a datanode hosting a block that the client is trying to read. > After too many of these failures (default 3, but configurable), the HDFS > client aborts the read and this bubbles up to the caller with the "could > not obtain block" error. > > I recommend troubleshooting this as a network connectivity issue. This > wiki page includes a few tips as a starting point: > > http://wiki.apache.org/hadoop/TroubleShooting > > Hope this helps, > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > > On Fri, Oct 25, 2013 at 4:53 AM, Divya R <avyakr...@gmail.com> wrote: > > > Hi Guys, > > > > I'm indexing data (~50 -100GB per day) from hadoop. Hadoop is Running > in > > cluster mode (having 2 dataNaodes currently). After every two or three > > hours I'm getting this exception. But both Data nodes are up and running. > > Can any one please guide me as to what I should do or If I'm doing > wrong. > > > > Code Snippet: > > public InitHadoop() { > > > > configuration = new Configuration(); > > configuration.set("fs.default.name", "hdfs://<<namenode > > IP>>:54310"); // Is this write to specify on namenode IP.? > > configuration.set("mapred.job.tracker", "hdfs://<<namenode > > IP>>:54311"); > > > > try { > > fileSystem = FileSystem.get(configuration); > > } catch (IOException e) { > > e.printStackTrace(); > > } > > } > > private void indexDocument(FSDataInputStream file) { > > > > Scanner scanner = new Scanner(file); > > > > while (scanner.hasNext() != null) { > > // Indexing code > > } > > } > > } > > > > Logs: > > > > 2013-10-25 09:37:57 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:57 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:57 INFO DFSClient:2432 - Could not obtain block > > blk_-8795538519317154213_432897 from any node: java.io.IOException: No > live > > nodes contain current block. Will get new block locations from namenode > and > > retry... > > 2013-10-25 09:37:58 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:58 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:58 INFO DFSClient:2432 - Could not obtain block > > blk_-5974673190155585497_432671 from any node: java.io.IOException: No > live > > nodes contain current block. Will get new block locations from namenode > and > > retry... > > 2013-10-25 09:37:59 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:59 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:59 INFO DFSClient:2432 - Could not obtain block > > blk_-1662761320365439855_431653 from any node: java.io.IOException: No > live > > nodes contain current block. Will get new block locations from namenode > and > > retry... > > 2013-10-25 09:37:59 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:59 WARN DFSClient:2266 - Failed to connect to > > /<<IP>>:50010, add to deadNodes and continuejava.net.BindException: > Cannot > > assign requested address > > 2013-10-25 09:37:59 WARN DFSClient:2400 - DFS Read: java.io.IOException: > > Could not obtain block: blk_8826777676488299245_432528 > > file=/flume/<<File.Name>>.1382639351042 > > at > > > > > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:2426) > > at > > > > > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2218) > > at > > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2381) > > at java.io.DataInputStream.read(DataInputStream.java:149) > > at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) > > at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) > > at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) > > at java.io.InputStreamReader.read(InputStreamReader.java:184) > > > > Regards, > > -Divya > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
