Did you try the setting I suggested? There is/was a known bug in HDFS which can cause issues which may include "abandoned" sockets such as you are describing.
-ryan On Mon, Jul 19, 2010 at 2:13 AM, Thomas Downing <[email protected]> wrote: > Thanks for the response, but my problem is not with FIN_WAIT2, it > is with FIN_WAIT1. > > If it was FIN_WAIT2, the only concern would be socket leakage, > and if setting the time out solved the issue, that would be great. > > The problem with FIN_WAIT1 is twofold - first, it is incumbent on > the application to notice and handle this problem; from the TCP stack > point of view, there is nothing wrong. It is just a special case of slow > consumer. The other problem is that it implies that something will be > lost if the socket is abandoned, there is data in the send queue of the > socket in FIN_WAIT1 that has not yet been delivered to the peer. > > On 7/16/2010 3:56 PM, Ryan Rawson wrote: >> >> I've been running with this setting on both the HDFS side and the >> HBase side for over a year now, it's a bit of voodoo but you might be >> running into well known suckage of HDFS. Try this one and restart >> your hbase& hdfs. >> >> The FIN_WAIT2/TIME_WAIT happens more on large concurrent gets, not so >> much for inserts. >> >> <property> >> <name>dfs.datanode.socket.write.timeout</name> >> <value>0</value> >> </property> >> >> -ryan >> >> >> On Fri, Jul 16, 2010 at 9:33 AM, Thomas Downing >> <[email protected]> wrote: >> >>> >>> Thanks for the response. >>> >>> My understanding is that TCP_FIN_TIMEOUT affects only FIN_WAIT2, >>> my problem is with FIN_WAIT1. >>> >>> While I do see some sockets in TIME_WAIT, they are only a few, and the >>> number is not growing. >>> >>> On 7/16/2010 12:07 PM, Hegner, Travis wrote: >>> >>>> >>>> Hi Thomas, >>>> >>>> I ran into a very similar issue when running slony-I on postgresql to >>>> replicate 15-20 databases. >>>> >>>> Adjusting the TCP_FIN_TIMEOUT parameters for the kernel may help to slow >>>> (or hopefully stop), the leaking sockets. I found some notes about >>>> adjusting >>>> TCP parameters here: >>>> http://www.hikaro.com/linux/tweaking-tcpip-syctl-conf.html >>>> >>>> > > [snip] >
