In DN logs, There are a lot of "480000 millis timeout" exceptions when some
scans finished, But Region server have no exceptions.
I analyzed the flow of HBase and Hadoop. I found we are using API
"readBuffer(byte buf[], int off, int len)", It means we need read the whole
block data, if we stop reading, some timeout will happen in DN side.
I want to know whether is the bad effect?
For example leaking the socket, I found some "CLOSE_WAIT" in our machine, I did
not know whether it is the result.
Who has experience? Please help explain.
Thank You.
The HDFS API
2012-01-07 07:02:40,753 INFO org.apache.hadoop.hdfs.DFSClient: Hbase is
invoking .................... seek(long targetPos)
2012-01-07 07:02:40,753 INFO org.apache.hadoop.hdfs.DFSClient: Hbase is
invoking.................................read(byte buf[], int off, int len)
2012-01-07 07:02:40,754 INFO org.apache.hadoop.hdfs.DFSClient: Hbase is
invoking.................................readBuffer(byte buf[], int off, int
len)
DN Logs:
2012-01-08 02:52:32,969 WARN datanode.DataNode
(DataXceiver.java:readBlock(274)) - DatanodeRegistration(158.1.130.33:10010,
storageID=DS-1985031385-158.1.130.33-10010-1318824003883, infoPort=10075,
ipcPort=10020):Got exception while serving blk_1325150650461_45153 to
/158.1.130.33:
java.net.SocketTimeoutException: 480000 millis timeout while waiting for
channel to be ready for write. ch : java.nio.channels.SocketChannel[connected
local=/158.1.130.33:10010 remote=/158.1.130.33:53792]
at
org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:247)
at
org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:410)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:508)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:247)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:130)
at java.lang.Thread.run(Thread.java:662)
2012-01-08 02:52:32,969 ERROR datanode.DataNode (DataXceiver.java:run(183)) -
org.apache.hadoop.hdfs.server.datanode.DataNode (158.1.130.33:10010,
storageID=DS-1985031385-158.1.130.33-10010-1318824003883, infoPort=10075,
ipcPort=10020):DataXceiver
java.net.SocketTimeoutException: 480000 millis timeout while waiting for
channel to be ready for write. ch : java.nio.channels.SocketChannel[connected
local=/158.1.130.33:10010 remote=/158.1.130.33:53792]
at
org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:247)
at
org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:410)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:508)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:247)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:130)
at java.lang.Thread.run(Thread.java:662)
