Hello Florian, this looks indeed like a resource exhaustion, probably related to some interaction between different modules.
If it happened only one time, it might be difficult to reproduce it. If it happens frequently, it probably makes sense to investigate more into the direction of the siptrace module. Cheers, Henning From: Floimair Florian via sr-users <[email protected]> Sent: Dienstag, 12. November 2024 11:07 To: Kamailio (SER) - Users Mailing List <[email protected]> Cc: Floimair Florian <[email protected]> Subject: [SR-Users] WG: [External] RE: Question regarding error message "failed to add new socket to the fd list" Thank you Henning! Here are the configured limits for Kamailio Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 31543 31543 processes Max open files 16384 16384 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 31543 31543 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us This is what I know so far: There seems to be a problem with siptrace module (going via TCP to a heplify-server on another host), but Iām not sure if this is a coincidence or the cause of the problems. I see quite a bit of these before the actual problem arises: Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322822]: ERROR: <core> [core/tcp_main.c:630]: _wbufq_add(): (591 bytes): write queue full or timeout (32498, total 43196, last write 0 s ago) Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322822]: ERROR: siptrace [../../core/forward.h:261]: msg_send_buffer(): tcp_send failed Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322822]: ERROR: siptrace [siptrace_hep.c:229]: trace_send_hep3_duplicate(): cannot send hep duplicate message Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: ERROR: <core> [core/tcp_main.c:4023]: handle_ser_child(): received CON_ERROR for 0x7fe054f8ae10 (id 727321), refcnt 2, flags 0x3096 At some point then all things go south and the log is flooded with these messages Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 2244, not in [0, 482) Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: ERROR: <core> [core/tcp_main.c:4677]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fe054f8ae10, fd 2244 And then ultimately the above combined with the initially mentioned Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: CRITICAL: <core> [core/tcp_main.c:4264]: handle_ser_child(): failed to add new socket to the fd list And occasionally also something like this Nov 4 07:54:01 kamailio-prod-westeurope-2 /usr/sbin/kamailio[322829]: ERROR: <core> [core/io_wait.h:373]: io_watch_add(): trying to overwrite entry 2247 watched for 5 in the hash 0x5621dc404ee0 (fd:-601539712, type:22049, data:0x5621dc23c004) with (2247, 2, 0x7fe04f85c2a0) In total these add up to roughly 3600 lines per second (!!) in the log, so it is quickly flooded with these. Florian FLOIMAIR Software Development - Symphony Cloud Services (1568) Von: Henning Westerholt <[email protected]<mailto:[email protected]>> Datum: Montag, 11. November 2024 um 19:52 An: Kamailio (SER) - Users Mailing List <[email protected]<mailto:[email protected]>> Cc: Floimair Florian <[email protected]<mailto:[email protected]>> Betreff: [External] RE: Question regarding error message "failed to add new socket to the fd list" CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hello Florian, Any other error messages before that error happens, e.g. something about the memory. Otherwise, it could be indeed a ulimit issue. First step would be to check the actual configured limits for the kamailio user. Cheers, Henning From: Floimair Florian via sr-users <[email protected]<mailto:[email protected]>> Sent: Montag, 11. November 2024 17:45 To: [email protected]<mailto:[email protected]> Cc: Floimair Florian <[email protected]<mailto:[email protected]>> Subject: [SR-Users] Question regarding error message "failed to add new socket to the fd list" Hi! We have recently had issues with one of our Production Kamailios. When those happened, the log was filled with the following message: CRITICAL: <core> [core/tcp_main.c:4528]: handle_new_connect(): failed to add new socket to the fd list Now I wonder what the best approach is to prevent this. We are using TCP/TLS only and I think this might be related to the file ulimit, but I am not sure about that. Shared memory is set to 512MB Can you give me a hint on what to look for? Thank you very much! P.S.: Sorry, I accidentally replied to a previous post of a different topic before which is totally unrelated (I think I should stop working for today š) FLORIAN FLOIMAIR Software Development - Symphony Cloud Services Commend International GmbH Saalachstrasse 51 5020 Salzburg, Austria [signature_2072127332] commend.com LG Salzburg / FN 178618z
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- [email protected] To unsubscribe send an email to [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender!
