Hi all,
We have observed several connections between nodes in CLOSE_WAIT after
several hours of operation:
At node 87:
netstat -tn | grep 7000
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:57625
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:51541
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:58447
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:51313
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:52065
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:58218
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:54986 ::ffff:192.168.2.88:7000
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:48272
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:55433
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:59138 ::ffff:192.168.2.88:7000
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:39074
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:59088
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:34012
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:55806
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:42472
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:45033
CLOSE_WAIT
At the other node: 88
netstat -tn | grep 7000
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:59138
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:46143
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:38202
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:55852
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:39208
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:55378
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:51061
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:44911
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:58447 ::ffff:192.168.2.87:7000
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:59614
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:35033
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:39074 ::ffff:192.168.2.87:7000
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:54986
ESTABLISHED
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:54772
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:39925
CLOSE_WAIT
tcp 0 0 ::ffff:192.168.2.88:7000 ::ffff:192.168.2.87:38124
CLOSE_WAIT
the setup only uses two nodes, replication factor = 2 with latest jdk 6u20
and cassandra 0.6.0
Afaik CLOSE_WAIT indicates there are opened sockets do not close properly.
Is anyone experience similar problem ? How do I do to find the root cause ?
Any help is appreciated.
