What is your file descriptor limit? If it is set to the 1024, you will want to up that considerably. Don't be afraid to go to 64k.
On Thu, Jul 15, 2010 at 3:38 AM, Peter Falk <[email protected]> wrote: > Hi, > > We are noticing the following errors in our datanode logs. We are running > hbase on top of hdfs, but are not noticing any errors in the hbase logs. So > it seems like the hdfs clients are not suffering from these errors. > However, > it would be nice to understand why they appear. We have upped the number of > xcievers to 1024 and are having some throughts about there are too many > sockets and some of them are timing out because they are not being used? > Also we have set dfs.datanode.socket.write.timeout and dfs.socket.timeout > to > about 10 minutes. Anyone knows why these errors appears, and perhaps how to > get rid of them? > > 2010-07-15 12:31:02,307 WARN > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 192.168.10.54:50010, > storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075, > ipcPort=50020):Got exception while serving blk_3305049902326993023_98358 to > /192.168.10.54: > java.net.SocketTimeoutException: 600000 millis timeout while waiting for > channel to be ready for write. ch : > java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010 > remote=/ > 192.168.10.54:46276] > at > > org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) > at > > org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159) > at > > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198) > at > > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313) > at > > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400) > at > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95) > at java.lang.Thread.run(Thread.java:619) > > 2010-07-15 12:31:02,308 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 192.168.10.54:50010, > storageID=DS-1341800526-192.168.10.54-50010-1278528499852, infoPort=50075, > ipcPort=50020):DataXceiver > java.net.SocketTimeoutException: 600000 millis timeout while waiting for > channel to be ready for write. ch : > java.nio.channels.SocketChannel[connected local=/192.168.10.54:50010 > remote=/ > 192.168.10.54:46276] > at > > org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) > at > > org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159) > at > > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198) > at > > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313) > at > > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400) > at > > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:180) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95) > at java.lang.Thread.run(Thread.java:619) > > TIA, > Peter >
