Hello, as you said it shows a single connections, I went to the code and I discovered a bug in exporting rpc command core.tcp_info, because it was missing the option that it returns an array. I fixed it in master branch with next commit:
- https://github.com/kamailio/kamailio/commit/24ca2e7760a8dada433b188348c768e7e224f10d and I will backport to stable branches. Meanwhile, you can use: kamcmd core.tcp_list which is not strict in validating the binrcp/jsonrpc response and eventually it will print all the tcp connections. Can you test that? Cheers, Daniel On 17.04.19 15:42, Aymeric Moizard wrote: > Hi Daniel, > > Tks for answering! Unfortunatly, core.tcp_list is only returning one > connection. > But core.tcp_info reports 184 opened connections (same for > "kamctl stats tcp") > > sudo kamctl rpc core.tcp_list > { > "jsonrpc": "2.0", > "result": { > "id": 439290, > "type": "TCP", > "state": "CONN_ACCEPT", > "timeout": 2567, > "lifetime": 3600, > "ref_count": 1, > "src_ip": "41.46.4.235", > "src_port": 4957, > "dst_ip": "91.121.30.149", > "dst_port": 5060 > }, > "id": 9158 > } > > sudo kamctl rpc core.tcp_info > { > "jsonrpc": "2.0", > "result": { > "readers": 16, > "max_connections": 50000, > "max_tls_connections": 50000, > "opened_connections": 184, > "opened_tls_connections": 64, > "write_queued_bytes": 0 > }, > "id": 9523 > } > > Did I missed something? > > > jack@sip:~$ /usr/sbin/kamailio -v > version: kamailio 5.2.2 (x86_64/linux) > flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, > USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, > SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, > USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, > USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES > ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE > 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > id: unknown > compiled with gcc 6.3.0 > > Regards > Aymeric > > Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla > <[email protected] <mailto:[email protected]>> a écrit : > > Hello, > > On 26.03.19 17:16, Aymeric Moizard wrote: >> Hi Again, >> >> Here is an issue with TCP connection being kept for more: >> >> Yesterday, I have discovered that a User-Agent (<Avaya IP Phone >> 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was >> sending REGISTER over new established TCP socket *every 2 seconds*. >> >> All the REGISTER was rejected with 401. (may be the device was >> misconfigured? or not receiving any of my answer? I can't tell) >> >> NOTE: You can see the expires header was very large: 86400, ie: >> 24 hours... >> >> I was checking the TCP/TLS connections on my server and >> discovered more than 1000 TCP established connection to that >> user/ip, and thus, I have tried to understand what happened. >> >> Checking the logs, I received 4855 REGISTER from this device from >> "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx >> one new TCP connection every 2.5 seconds... >> >> Today, I decided to check it again around 11am. >> >> jack@sip:~$ sudo kamctl stats tcp >> { >> "jsonrpc": "2.0", >> "result": [ >> "tcp:con_reset = 1857", >> "tcp:con_timeout = 35927", >> "tcp:connect_failed = 25", >> "tcp:connect_success = 2", >> "tcp:current_opened_connections = 2291", >> "tcp:current_write_queue_size = 0", >> "tcp:established = 80778", >> "tcp:local_reject = 0", >> "tcp:passive_open = 80776", >> "tcp:send_timeout = 2", >> "tcp:sendq_full = 0" >> ], >> "id": 7305 >> } >> >> There was still A LOT of established connections. And the >> connections have been established more than 24 hours ago. >> >> At 11H16: >> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | >> grep ESTA | wc -l >> 1161 >> At 11H22: >> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | >> grep ESTA | wc -l >> 1018 >> At 11H35: >> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | >> grep ESTA | wc -l >> 655 >> At 13H >> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | >> grep ESTA | wc -l >> 0 >> >> So the established connections are all gone now. >> >> Between 11h16 and 11H35, I was seeing the server regularly >> sending [FIN, ACK] over each TCP established connection, with >> retransmissions for all of them. (no incoming trafic) >> >> I do not have numbers/capture/stats, but I think that kamailio >> was already closing some >> connection yesterday. I don't know when kamailio started to try >> closing those connections. >> >> I'm now back with this status: >> >> At 13pm: >> jack@sip:~$ sudo kamctl stats tcp >> { >> "jsonrpc": "2.0", >> "result": [ >> "tcp:con_reset = 1896", >> "tcp:con_timeout = 38042", >> "tcp:connect_failed = 26", >> "tcp:connect_success = 2", >> "tcp:current_opened_connections = 939", >> "tcp:current_write_queue_size = 0", >> "tcp:established = 81950", >> "tcp:local_reject = 0", >> "tcp:passive_open = 81948", >> "tcp:send_timeout = 2", >> "tcp:sendq_full = 0" >> ], >> "id": 12734 >> } >> >> With around 155 registration entries using TCP and TLS in my >> location database. >> >> As you can see, tcp:current_opened_connections = 939 is still >> pretty high compared to >> my currently registred users. >> >> I have "modparam("registrar", "max_expires", 86400)", because I'm >> keeping contact entries (even with TCP connection down) for push >> notifications. >> >> I have "tcp_connection_lifetime=3600" configured. >> >> Question 1 >> >> With "tcp_connection_lifetime=3600", I would expect kamailio to >> close the established connection after 3600 seconds without >> traffic. It is pretty obvious that no data has been exchanged >> over the 4855 established connection during a day. >> >> Despite the issue with the Avaya phones is solved automatically >> after a day, I guess similar stuff or happening, at a different >> rate, for other users as well. (because >> current_opened_connections is way higher than registred TCP/TLS >> users) > > > Yes, tcp connections should be closed if no traffic on them for > the lifetime duration. > > >> >> Question 2 >> >> I can list TLS connection with "kamctl rpc tls.list" >> Can I get a similar list for TCP? (lsof returns a lot of >> duplicates...) > > > Yes, see: > > > http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list > > Maybe you can compare what is listed by the rpc command to see > what kamailio actually sees as active connections. > > Cheers, > Daniel > > -- > Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> > www.twitter.com/miconda <http://www.twitter.com/miconda> -- > www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> > Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com > <http://www.kamailioworld.com> > > > > -- > Antisip - http://www.antisip.com -- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
_______________________________________________ Kamailio (SER) - Users Mailing List [email protected] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
