[
https://issues.apache.org/jira/browse/CASSANDRA-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13807912#comment-13807912
]
Prateek commented on CASSANDRA-6259:
------------------------------------
Sylvain,
The cascading client uses CqlRecordReader (not ColumnFamilyInputFormat).
It uses native transport (port 9042) for cql as shown in the description. The
problem is 100% reproducible. The client is well-behaved since it closes the
connection (CLOSE_WAIT Indicates that the server has received the first FIN
signal from the client and the connection is in the process of being closed).
Try putting some load on the server to reproduce the problem. I only see error
messages in the server log that indicate too many open file descriptors. Should
I be looking for something in particular?
Let me know if I can help.
Thanks,
Prateek
> Cassandra 2.0.1 server has too many tcp connections in CLOSE_WAIT
> -----------------------------------------------------------------
>
> Key: CASSANDRA-6259
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6259
> Project: Cassandra
> Issue Type: Bug
> Reporter: Prateek
> Assignee: Sylvain Lebresne
>
> We are using cassandra 2.0.1 server with cascading client. The cassandra tap
> used is https://github.com/ifesdjeen/cascading-cassandra (1.0.0-rc6). The
> problem arises after the server is running for a few days. The server has
> 100,000+ connections in tcp CLOSE_WAIT state and cannot accept any more
> connections. All map reduce jobs start failing. This seems to be a bug with
> cassandra 2.0.1 server not closing connections properly.
> [(bloomreach-ami) ubuntu@ip-10-91-15-6 :/mnt/cassandra/data]# lsof -n | grep
> java | grep CLOSE_WAIT | wc -l
> 116321
> java 25427 ubuntu *537u IPv4 9337512 0t0
> TCP 10.91.15.6:9042->10.171.11.168:34217 (CLOSE_WAIT)
> java 25427 ubuntu *540u IPv4 9107933 0t0
> TCP 10.91.15.6:9042->10.92.99.19:45820 (CLOSE_WAIT)
> java 25427 ubuntu *543u IPv4 9110100 0t0
> TCP 10.91.15.6:9042->10.86.106.249:47585 (CLOSE_WAIT)
> java 25427 ubuntu *544u IPv4 9110072 0t0
> TCP 10.91.15.6:9042->10.86.106.249:47364 (CLOSE_WAIT)
> java 25427 ubuntu *546u IPv4 9110110 0t0
> TCP 10.91.15.6:9042->10.92.99.19:46162 (CLOSE_WAIT)
> java 25427 ubuntu *547u IPv4 9110093 0t0
> TCP 10.91.15.6:9042->10.86.106.249:47518 (CLOSE_WAIT)
> java 25427 ubuntu *548u IPv4 9337583 0t0
> TCP 10.91.15.6:9042->10.171.11.168:34361 (CLOSE_WAIT)
> java 25427 ubuntu *549u IPv4 9110114 0t0
> TCP 10.91.15.6:9042->10.92.99.19:46212 (CLOSE_WAIT)
> java 25427 ubuntu *551u IPv4 9110117 0t0
--
This message was sent by Atlassian JIRA
(v6.1#6144)
